- Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Exploring Heap Size Limits

by phil＠highoctane.be

I did some more experiments and that's the most I can get from the script, no matter how much free memory there is on the machine. Adding ./pharo --memory 2000m ... has no effect. [PhilMac:~/Documents/Smalltalk/2-MyWorkspaces/workspaceMemTest philippeback$] ./pharo Pharo.image memtest.st 501 1024000 memtest.st uptime 0h0m0s memory 25,004,444 bytes old 19,213,688 bytes (76.80000000000001%) young 5,384 bytes (0.0%) used 19,219,072 bytes (76.9%) free 5,785,372 bytes (23.1%) GCs 5 (62ms between GCs) full 1 totalling 76ms (24.5% uptime), avg 76.0ms incr 4 totalling 10ms (3.2% uptime), avg 2.5ms tenures 0 allocating: 501 times 1024000 bytes allocated: 513.02 MB time to run 0:00:00:06.27 3 times GC time to run 0:00:00:00.494 uptime 0h0m7s memory 534,974,360 bytes old 532,256,456 bytes (99.5%) young 7,092 bytes (0.0%) used 532,263,548 bytes (99.5%) free 2,710,812 bytes (0.5%) GCs 344 (21ms between GCs) full 87 totalling 5,822ms (82.2% uptime), avg 66.9ms incr 257 totalling 659ms (9.3% uptime), avg 2.6ms tenures 0 Since last view 339 (20ms between GCs) uptime 6.800000000000001s full 86 totalling 5,746ms (84.9% uptime), avg 66.8ms incr 253 totalling 649ms (9.600000000000001% uptime), avg 2.6ms tenures 0

June 29, 2013

Re: [Pharo-dev] Exploring Heap Size Limits

by phil＠highoctane.be

On OSX Lion w/ 8GB RAM, the first call gives me: [PhilMac:~/Documents/Smalltalk/2-MyWorkspaces/workspaceMemTest philippeback$] ./run.sh memtest.st uptime 0h0m0s memory 25,004,440 bytes old 19,213,728 bytes (76.80000000000001%) young 5,384 bytes (0.0%) used 19,219,112 bytes (76.9%) free 5,785,328 bytes (23.1%) GCs 5 (127ms between GCs) full 1 totalling 72ms (11.4% uptime), avg 72.0ms incr 4 totalling 11ms (1.7000000000000002% uptime), avg 2.8000000000000003ms tenures 0 allocating: 512 times 1024000 bytes =============================================================================== Notice: Errors in script loaded from /Users/philippeback/Documents/Smalltalk/2-MyWorkspaces/workspaceMemTest/ memtest.st =============================================================================== Errors in script loaded from /Users/philippeback/Documents/Smalltalk/2-MyWorkspaces/workspaceMemTest/ memtest.st ==== Startup Error: OutOfMemory ByteArray class(Behavior)>>basicNew: ByteArray class(Behavior)>>new: UndefinedObject>>DoIt in Block: [t6... SmallInteger(Integer)>>timesRepeat: UndefinedObject>>DoIt in Block: [:t6 | t1... Array class(SequenceableCollection class)>>new:streamContents: UndefinedObject>>DoIt in Block: [t4 := Array... Time class>>millisecondsToRun: BlockClosure>>timeToRun UndefinedObject>>DoIt Compiler>>evaluate:in:to:notifying:ifFail:logged: Compiler class>>evaluate:for:notifying:logged: Compiler class>>evaluate:for:logged: Compiler class>>evaluate:logged: DoItDeclaration>>import CodeImporter>>evaluate in Block: [:decl | value := decl import] OrderedCollection>>do: CodeImporter>>evaluate BasicCodeLoader>>installSourceFile: in Block: [codeImporter evaluate] BlockClosure>>on:do: BasicCodeLoader>>handleErrorsDuring:reference: BasicCodeLoader>>installSourceFile: BasicCodeLoader>>installSourceFiles in Block: [:reference | self installSourceFile: reference] OrderedCollection>>do: BasicCodeLoader>>installSourceFiles in Block: [sourceFiles... BlockClosure>>ensure: BasicCodeLoader>>installSourceFiles BasicCodeLoader>>activate BasicCodeLoader class(CommandLineHandler class)>>activateWith: DefaultCommandLineHandler>>handleSubcommand Got startup errors: OutOfMemory Now, I did a purge. Same results. Need to pass a parameter or something? This is on a fresh 2.0 + vmLatest Phil

June 29, 2013

Re: [Pharo-dev] VM ppa question

by Damien Cassou

On Sat, Jun 29, 2013 at 11:39 AM, GOUBIER Thierry <thierry.goubier(a)cea.fr> wrote: > I believe Damien said a while ago to switch to an "official' pharo ppa instead of his: > > deb http://ppa.launchpad.net/pharo/stable/ubuntu raring main I think you are right. Users need to delete the old PPA reference (in /etc/apt/sources.list.d) and add the new one http://www.pharo.org/pharo-download/ubuntu -- Damien Cassou http://damiencassou.seasidehosting.st "Success is the ability to go from one failure to another without losing enthusiasm." Winston Churchill

June 29, 2013

Re: [Pharo-dev] Question about closure compilation

by Marcus Denker

On Jun 29, 2013, at 11:47 AM, Norbert Hartl <norbert(a)hartl.name> wrote: >> > > Thanks. So please let us separate problems. In the example from Sven there are two problems: > > - increase of memory per function invocation. That problem we don't have Yes, you are right. I am not really knowledgeable about JavaScriptâ¦ especially the strange stuff in there. ;-) > - easily keeping a reference to memory preventing it from being garbage collected. That is what you are talking about. And to me there is nothing wrong with it > Yes. It could lead to an object not being GCed, but this should not be problem in practice. Marcus

June 29, 2013

Re: [Pharo-dev] Question about closure compilation

by Norbert Hartl

Am 29.06.2013 um 11:12 schrieb Marcus Denker <marcus.denker(a)inria.fr>: >> >> OK, that seems OK. >> >> The reason I was asking was because I wanted to know if this JavaScript bug would hit us as well: >> >> http://point.davidglasser.net/2013/06/27/surprising-javascript-memory-leak.… >> >> It is slightly more complicated than the basic question of closing over an unreferenced variable though. >> >> I just can't figure out a way to quickly test it in Pharo. I think it is worth looking into. > > Yes, we have exactly the same problem. > > imagine a method like this: > > test > | bigObject smallObject | > > [bigObject := Array new: 1000000] value. > ^[smallObject := 1]. > > > both big and small are escaping variables that are written. This means they are allocated > in an Array on the heap (Cog is calling this a TempVector). > > There is just one tempVector created per scope, so we will have a tempVector with both > smallObject and bigObject inside created for the method. This then is passed to the blocks, which > access the temps via the vector. > > The bytecode looks like this: > > 21 <8A 02> push: (Array new: 2) > 23 <68> popIntoTemp: 0 > 24 <10> pushTemp: 0 > 25 <8F 10 00 07> closureNumCopied: 1 numArgs: 0 bytes 29 to 35 > 29 <40> pushLit: Array > 30 <21> pushConstant: 1000000 > 31 <CD> send: new: > 32 <8D 00 00> storeIntoTemp: 0 inVectorAt: 0 > 35 <7D> blockReturn > 36 <C9> send: value > 37 <87> pop > 38 <10> pushTemp: 0 > 39 <8F 10 00 05> closureNumCopied: 1 numArgs: 0 bytes 43 to 47 > 43 <76> pushConstant: 1 > 44 <8D 01 00> storeIntoTemp: 1 inVectorAt: 0 > 47 <7D> blockReturn > 48 <7C> returnTop > > The closure [smallObject := 1] now, even though not referencing the largeObject, references > the tempVector. This tempVector holds on the to large object --> memory is wasted that is not needed. > > Back when implementing the SemChecker, I was thinking about this and wonderingâ¦ for sure > this can be optimized by allocating multiple arrays per scope, depending on how they are used later. > But even then I am sure you can construct strange cases (multiple nested blocks). > > And of course allocating 2 Arrays (or more) wastes memory, too. > > So this is not a simple problemâ¦ the question is how much it matters in practice, though. Thanks. So please let us separate problems. In the example from Sven there are two problems: - increase of memory per function invocation. That problem we don't have - easily keeping a reference to memory preventing it from being garbage collected. That is what you are talking about. And to me there is nothing wrong with it Norbert

June 29, 2013

Re: [Pharo-dev] Question about closure compilation

by Norbert Hartl

Am 28.06.2013 um 23:09 schrieb Sven Van Caekenberghe <sven(a)stfx.eu>: > > On 28 Jun 2013, at 22:42, Marcus Denker <marcus.denker(a)inria.fr> wrote: > >> >> On Jun 28, 2013, at 9:26 PM, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >> >>> Hi, >>> >>> Does the closure returned from >>> >>> | foo | >>> foo := #( data 1 2 3 ). >>> [ :x | x + 1 ] >>> >>> close over foo ? >> No. >> >>> Since foo is not used, it should not, right ? >> right. if we get the AST of your method >> >> >> ast := (TT>>#tt) ast. >> (ast scope lookupVar: 'foo') isEscaping >> >> --> false. > > OK, that seems OK. > > The reason I was asking was because I wanted to know if this JavaScript bug would hit us as well: > > http://point.davidglasser.net/2013/06/27/surprising-javascript-memory-leak.… > > It is slightly more complicated than the basic question of closing over an unreferenced variable though. > > I just can't figure out a way to quickly test it in Pharo. I think it is worth looking into. I don't think it is easy to do in pharo. The problem in the article only works because a string primitive is directly referenced. Primitives are copied as a whole when assigned. An object holding that huge string wouldn't be a problem because the enclosed activation record would just hold a reference to that object. As we don't have primitive types in smalltalk the problem doesn't apply IMHO. So his example is the combination that on every function invocation an activation record is created and the primitive string is copied into. The chain of the calling activation records _is_ the lexical scope IIRC. So tweaking the enclosed environment would need to copy and alter the activation records. And to be honest I fail to see why having foo in the scope is wrong. I mean why is it an error if it is there? In my opinion it is normal that foo is there but it can be an optimization to get rid of it, no? And I'm not sure it is easy in most cases to determine the liveliness of a variable. Norbert > >> Now the question is what the inspector of the context should show? >> >> e.g. >> >> tt >> | foo | >> foo := #( data 1 2 3 ). >> ^[ :x | x + 1. thisContext inspect ] >> >> >> if we now execute: >> >> TT new tt value: 2 >> >> the inspector pops up and shows both x and foo. >> >> The reason is that we need to decide what to show >> >> 1) the variable that is stored in the context >> 2) all *possible* variables that I can write in the block and recompile and it works. >> >> 1) is this: >> ast statements last value scope localTempNames >> >> 2) is this: >> >> ast statements last value scope allTemps >> >> Opal implements is like this because the old Compiler (DebuggerMethodMap) did it like this. >> At first I thought this is wrong and I wanted to change it. >> But then I thought that it is not that wrong: in the debugger the inspector of the context shows >> not just the vars that are in the context, but all that *could* be there. All where I can lookup and >> get a result, e.g. >> >> ast statements last value scope lookupVar: 'foo' >> >> Thus all those that I *could* write in the code inside the block. >> >> I am not sure, thoughâ¦ maybe we should change it. >> >> Marcus >> >> >>> But it seems that the outer context of the block closure knows about foo, although the value seems nil. >>> >>> And if it closes over foo, foo's value/binding won't be GC-ed so long as there is a reference to the block closure, even though foo is not practically accessible. >>> >>> Sven > >

June 29, 2013

Re: [Pharo-dev] VM ppa question

by GOUBIER Thierry

Hi Sven, I believe Damien said a while ago to switch to an "official' pharo ppa instead of his: deb http://ppa.launchpad.net/pharo/stable/ubuntu raring main (I still have both in my /etc/apt/sources.list.d/ :)) Thierry ________________________________________ De : Pharo-dev [pharo-dev-bounces(a)lists.pharo.org] de la part de Sven Van Caekenberghe [sven(a)stfx.eu] Date d'envoi : samedi 29 juin 2013 11:31 Ã : Pharo Development List Objet : [Pharo-dev] VM ppa question Hi, On a machine where I once installed the VM via the ppa method, I now get apt some errors: t3@coolerslave:smalltalk$ sudo apt-get update [sudo] password for t3: Get:1 http://security.ubuntu.com quantal-security Release.gpg [933 B] Hit http://be.archive.ubuntu.com quantal Release.gpg Ign http://ppa.launchpad.net quantal Release.gpg Get:2 http://security.ubuntu.com quantal-security Release [49.6 kB] Get:3 http://be.archive.ubuntu.com quantal-updates Release.gpg [933 B] Hit http://ppa.launchpad.net quantal Release.gpg Hit http://be.archive.ubuntu.com quantal-backports Release.gpg Hit http://ppa.launchpad.net quantal Release.gpg Ign http://ppa.launchpad.net quantal Release Hit http://be.archive.ubuntu.com quantal Release Get:4 http://be.archive.ubuntu.com quantal-updates Release [49.6 kB] Hit http://ppa.launchpad.net quantal Release Hit http://ppa.launchpad.net quantal Release Ign http://ppa.launchpad.net quantal/main Sources/DiffIndex Hit http://be.archive.ubuntu.com quantal-backports Release Get:5 http://security.ubuntu.com quantal-security/main Sources [51.9 kB] Ign http://ppa.launchpad.net quantal/main amd64 Packages/DiffIndex Hit http://be.archive.ubuntu.com quantal/main Sources Hit http://be.archive.ubuntu.com quantal/restricted Sources Hit http://be.archive.ubuntu.com quantal/universe Sources Hit http://be.archive.ubuntu.com quantal/multiverse Sources Ign http://ppa.launchpad.net quantal/main i386 Packages/DiffIndex Hit http://be.archive.ubuntu.com quantal/main amd64 Packages Get:6 http://security.ubuntu.com quantal-security/restricted Sources [1,833 B] Hit http://be.archive.ubuntu.com quantal/restricted amd64 Packages Get:7 http://security.ubuntu.com quantal-security/universe Sources [19.2 kB] Hit http://be.archive.ubuntu.com quantal/universe amd64 Packages Get:8 http://security.ubuntu.com quantal-security/multiverse Sources [696 B] Hit http://ppa.launchpad.net quantal/main Sources Hit http://be.archive.ubuntu.com quantal/multiverse amd64 Packages Get:9 http://security.ubuntu.com quantal-security/main amd64 Packages [146 kB] Hit http://ppa.launchpad.net quantal/main amd64 Packages Hit http://be.archive.ubuntu.com quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/restricted i386 Packages Hit http://ppa.launchpad.net quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/universe i386 Packages Get:10 http://security.ubuntu.com quantal-security/restricted amd64 Packages [3,469 B] Hit http://be.archive.ubuntu.com quantal/multiverse i386 Packages Get:11 http://security.ubuntu.com quantal-security/universe amd64 Packages [54.7 kB] Hit http://ppa.launchpad.net quantal/main Sources Get:12 http://security.ubuntu.com quantal-security/multiverse amd64 Packages [1,152 B] Hit http://be.archive.ubuntu.com quantal/main Translation-en Get:13 http://security.ubuntu.com quantal-security/main i386 Packages [144 kB] Hit http://ppa.launchpad.net quantal/main amd64 Packages Hit http://be.archive.ubuntu.com quantal/multiverse Translation-en Hit http://ppa.launchpad.net quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/restricted Translation-en Get:14 http://security.ubuntu.com quantal-security/restricted i386 Packages [3,531 B] Get:15 http://security.ubuntu.com quantal-security/universe i386 Packages [55.2 kB] Hit http://be.archive.ubuntu.com quantal/universe Translation-en Get:16 http://be.archive.ubuntu.com quantal-updates/main Sources [106 kB] Get:17 http://security.ubuntu.com quantal-security/multiverse i386 Packages [1,396 B] Hit http://security.ubuntu.com quantal-security/main Translation-en Get:18 http://be.archive.ubuntu.com quantal-updates/restricted Sources [2,564 B] Get:19 http://be.archive.ubuntu.com quantal-updates/universe Sources [88.8 kB] Hit http://security.ubuntu.com quantal-security/multiverse Translation-en Hit http://security.ubuntu.com quantal-security/restricted Translation-en Hit http://security.ubuntu.com quantal-security/universe Translation-en Get:20 http://be.archive.ubuntu.com quantal-updates/multiverse Sources [3,998 B] Get:21 http://be.archive.ubuntu.com quantal-updates/main amd64 Packages [273 kB] Ign http://security.ubuntu.com quantal-security/main Translation-en_US Get:22 http://be.archive.ubuntu.com quantal-updates/restricted amd64 Packages [4,804 B] Ign http://security.ubuntu.com quantal-security/multiverse Translation-en_US Get:23 http://be.archive.ubuntu.com quantal-updates/universe amd64 Packages [195 kB] Ign http://security.ubuntu.com quantal-security/restricted Translation-en_US Ign http://security.ubuntu.com quantal-security/universe Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en_US Get:24 http://be.archive.ubuntu.com quantal-updates/multiverse amd64 Packages [10.8 kB] Ign http://ppa.launchpad.net quantal/main Translation-en Get:25 http://be.archive.ubuntu.com quantal-updates/main i386 Packages [272 kB] Err http://ppa.launchpad.net quantal/main Sources 404 Not Found Err http://ppa.launchpad.net quantal/main amd64 Packages 404 Not Found Ign http://ppa.launchpad.net quantal/main Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en Ign http://ppa.launchpad.net quantal/main Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en Get:26 http://be.archive.ubuntu.com quantal-updates/restricted i386 Packages [4,841 B] Err http://ppa.launchpad.net quantal/main i386 Packages 404 Not Found Get:27 http://be.archive.ubuntu.com quantal-updates/universe i386 Packages [195 kB] Get:28 http://be.archive.ubuntu.com quantal-updates/multiverse i386 Packages [10.9 kB] Hit http://be.archive.ubuntu.com quantal-updates/main Translation-en Hit http://be.archive.ubuntu.com quantal-updates/multiverse Translation-en Hit http://be.archive.ubuntu.com quantal-updates/restricted Translation-en Hit http://be.archive.ubuntu.com quantal-updates/universe Translation-en Hit http://be.archive.ubuntu.com quantal-backports/main Sources Hit http://be.archive.ubuntu.com quantal-backports/restricted Sources Hit http://be.archive.ubuntu.com quantal-backports/universe Sources Hit http://be.archive.ubuntu.com quantal-backports/multiverse Sources Hit http://be.archive.ubuntu.com quantal-backports/main amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/restricted amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/universe amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/multiverse amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/main i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/restricted i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/universe i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/multiverse i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/main Translation-en Hit http://be.archive.ubuntu.com quantal-backports/multiverse Translation-en Hit http://be.archive.ubuntu.com quantal-backports/restricted Translation-en Hit http://be.archive.ubuntu.com quantal-backports/universe Translation-en Ign http://be.archive.ubuntu.com quantal/main Translation-en_US Ign http://be.archive.ubuntu.com quantal/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal/universe Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/main Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/universe Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/main Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/universe Translation-en_US Fetched 1,752 kB in 3s (439 kB/s) W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/source/Sour… 404 Not Found W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/binary-amd6… 404 Not Found W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/binary-i386… 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead. apt has always been a bit of mystery to me. Maybe the problem is that the ppa moved ? Is there a way to fix this ? Maybe uninstall some stuff ? Thx, Sven

June 29, 2013

VM ppa question

by Sven Van Caekenberghe

Hi, On a machine where I once installed the VM via the ppa method, I now get apt some errors: t3@coolerslave:smalltalk$ sudo apt-get update [sudo] password for t3: Get:1 http://security.ubuntu.com quantal-security Release.gpg [933 B] Hit http://be.archive.ubuntu.com quantal Release.gpg Ign http://ppa.launchpad.net quantal Release.gpg Get:2 http://security.ubuntu.com quantal-security Release [49.6 kB] Get:3 http://be.archive.ubuntu.com quantal-updates Release.gpg [933 B] Hit http://ppa.launchpad.net quantal Release.gpg Hit http://be.archive.ubuntu.com quantal-backports Release.gpg Hit http://ppa.launchpad.net quantal Release.gpg Ign http://ppa.launchpad.net quantal Release Hit http://be.archive.ubuntu.com quantal Release Get:4 http://be.archive.ubuntu.com quantal-updates Release [49.6 kB] Hit http://ppa.launchpad.net quantal Release Hit http://ppa.launchpad.net quantal Release Ign http://ppa.launchpad.net quantal/main Sources/DiffIndex Hit http://be.archive.ubuntu.com quantal-backports Release Get:5 http://security.ubuntu.com quantal-security/main Sources [51.9 kB] Ign http://ppa.launchpad.net quantal/main amd64 Packages/DiffIndex Hit http://be.archive.ubuntu.com quantal/main Sources Hit http://be.archive.ubuntu.com quantal/restricted Sources Hit http://be.archive.ubuntu.com quantal/universe Sources Hit http://be.archive.ubuntu.com quantal/multiverse Sources Ign http://ppa.launchpad.net quantal/main i386 Packages/DiffIndex Hit http://be.archive.ubuntu.com quantal/main amd64 Packages Get:6 http://security.ubuntu.com quantal-security/restricted Sources [1,833 B] Hit http://be.archive.ubuntu.com quantal/restricted amd64 Packages Get:7 http://security.ubuntu.com quantal-security/universe Sources [19.2 kB] Hit http://be.archive.ubuntu.com quantal/universe amd64 Packages Get:8 http://security.ubuntu.com quantal-security/multiverse Sources [696 B] Hit http://ppa.launchpad.net quantal/main Sources Hit http://be.archive.ubuntu.com quantal/multiverse amd64 Packages Get:9 http://security.ubuntu.com quantal-security/main amd64 Packages [146 kB] Hit http://ppa.launchpad.net quantal/main amd64 Packages Hit http://be.archive.ubuntu.com quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/restricted i386 Packages Hit http://ppa.launchpad.net quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/universe i386 Packages Get:10 http://security.ubuntu.com quantal-security/restricted amd64 Packages [3,469 B] Hit http://be.archive.ubuntu.com quantal/multiverse i386 Packages Get:11 http://security.ubuntu.com quantal-security/universe amd64 Packages [54.7 kB] Hit http://ppa.launchpad.net quantal/main Sources Get:12 http://security.ubuntu.com quantal-security/multiverse amd64 Packages [1,152 B] Hit http://be.archive.ubuntu.com quantal/main Translation-en Get:13 http://security.ubuntu.com quantal-security/main i386 Packages [144 kB] Hit http://ppa.launchpad.net quantal/main amd64 Packages Hit http://be.archive.ubuntu.com quantal/multiverse Translation-en Hit http://ppa.launchpad.net quantal/main i386 Packages Hit http://be.archive.ubuntu.com quantal/restricted Translation-en Get:14 http://security.ubuntu.com quantal-security/restricted i386 Packages [3,531 B] Get:15 http://security.ubuntu.com quantal-security/universe i386 Packages [55.2 kB] Hit http://be.archive.ubuntu.com quantal/universe Translation-en Get:16 http://be.archive.ubuntu.com quantal-updates/main Sources [106 kB] Get:17 http://security.ubuntu.com quantal-security/multiverse i386 Packages [1,396 B] Hit http://security.ubuntu.com quantal-security/main Translation-en Get:18 http://be.archive.ubuntu.com quantal-updates/restricted Sources [2,564 B] Get:19 http://be.archive.ubuntu.com quantal-updates/universe Sources [88.8 kB] Hit http://security.ubuntu.com quantal-security/multiverse Translation-en Hit http://security.ubuntu.com quantal-security/restricted Translation-en Hit http://security.ubuntu.com quantal-security/universe Translation-en Get:20 http://be.archive.ubuntu.com quantal-updates/multiverse Sources [3,998 B] Get:21 http://be.archive.ubuntu.com quantal-updates/main amd64 Packages [273 kB] Ign http://security.ubuntu.com quantal-security/main Translation-en_US Get:22 http://be.archive.ubuntu.com quantal-updates/restricted amd64 Packages [4,804 B] Ign http://security.ubuntu.com quantal-security/multiverse Translation-en_US Get:23 http://be.archive.ubuntu.com quantal-updates/universe amd64 Packages [195 kB] Ign http://security.ubuntu.com quantal-security/restricted Translation-en_US Ign http://security.ubuntu.com quantal-security/universe Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en_US Get:24 http://be.archive.ubuntu.com quantal-updates/multiverse amd64 Packages [10.8 kB] Ign http://ppa.launchpad.net quantal/main Translation-en Get:25 http://be.archive.ubuntu.com quantal-updates/main i386 Packages [272 kB] Err http://ppa.launchpad.net quantal/main Sources 404 Not Found Err http://ppa.launchpad.net quantal/main amd64 Packages 404 Not Found Ign http://ppa.launchpad.net quantal/main Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en Ign http://ppa.launchpad.net quantal/main Translation-en_US Ign http://ppa.launchpad.net quantal/main Translation-en Get:26 http://be.archive.ubuntu.com quantal-updates/restricted i386 Packages [4,841 B] Err http://ppa.launchpad.net quantal/main i386 Packages 404 Not Found Get:27 http://be.archive.ubuntu.com quantal-updates/universe i386 Packages [195 kB] Get:28 http://be.archive.ubuntu.com quantal-updates/multiverse i386 Packages [10.9 kB] Hit http://be.archive.ubuntu.com quantal-updates/main Translation-en Hit http://be.archive.ubuntu.com quantal-updates/multiverse Translation-en Hit http://be.archive.ubuntu.com quantal-updates/restricted Translation-en Hit http://be.archive.ubuntu.com quantal-updates/universe Translation-en Hit http://be.archive.ubuntu.com quantal-backports/main Sources Hit http://be.archive.ubuntu.com quantal-backports/restricted Sources Hit http://be.archive.ubuntu.com quantal-backports/universe Sources Hit http://be.archive.ubuntu.com quantal-backports/multiverse Sources Hit http://be.archive.ubuntu.com quantal-backports/main amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/restricted amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/universe amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/multiverse amd64 Packages Hit http://be.archive.ubuntu.com quantal-backports/main i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/restricted i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/universe i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/multiverse i386 Packages Hit http://be.archive.ubuntu.com quantal-backports/main Translation-en Hit http://be.archive.ubuntu.com quantal-backports/multiverse Translation-en Hit http://be.archive.ubuntu.com quantal-backports/restricted Translation-en Hit http://be.archive.ubuntu.com quantal-backports/universe Translation-en Ign http://be.archive.ubuntu.com quantal/main Translation-en_US Ign http://be.archive.ubuntu.com quantal/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal/universe Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/main Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal-updates/universe Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/main Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/multiverse Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/restricted Translation-en_US Ign http://be.archive.ubuntu.com quantal-backports/universe Translation-en_US Fetched 1,752 kB in 3s (439 kB/s) W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/source/Sour… 404 Not Found W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/binary-amd6… 404 Not Found W: Failed to fetch http://ppa.launchpad.net/cassou/pharo/ubuntu/dists/quantal/main/binary-i386… 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead. apt has always been a bit of mystery to me. Maybe the problem is that the ppa moved ? Is there a way to fix this ? Maybe uninstall some stuff ? Thx, Sven

June 29, 2013

Re: [Pharo-dev] Question about closure compilation

by Sven Van Caekenberghe

Hi Marcus, On 29 Jun 2013, at 11:12, Marcus Denker <marcus.denker(a)inria.fr> wrote: >>> >> >> OK, that seems OK. >> >> The reason I was asking was because I wanted to know if this JavaScript bug would hit us as well: >> >> http://point.davidglasser.net/2013/06/27/surprising-javascript-memory-leak.… >> >> It is slightly more complicated than the basic question of closing over an unreferenced variable though. >> >> I just can't figure out a way to quickly test it in Pharo. I think it is worth looking into. > > Yes, we have exactly the same problem. > > imagine a method like this: > > test > | bigObject smallObject | > > [bigObject := Array new: 1000000] value. > ^[smallObject := 1]. > > > both big and small are escaping variables that are written. This means they are allocated > in an Array on the heap (Cog is calling this a TempVector). > > There is just one tempVector created per scope, so we will have a tempVector with both > smallObject and bigObject inside created for the method. This then is passed to the blocks, which > access the temps via the vector. > > The bytecode looks like this: > > 21 <8A 02> push: (Array new: 2) > 23 <68> popIntoTemp: 0 > 24 <10> pushTemp: 0 > 25 <8F 10 00 07> closureNumCopied: 1 numArgs: 0 bytes 29 to 35 > 29 <40> pushLit: Array > 30 <21> pushConstant: 1000000 > 31 <CD> send: new: > 32 <8D 00 00> storeIntoTemp: 0 inVectorAt: 0 > 35 <7D> blockReturn > 36 <C9> send: value > 37 <87> pop > 38 <10> pushTemp: 0 > 39 <8F 10 00 05> closureNumCopied: 1 numArgs: 0 bytes 43 to 47 > 43 <76> pushConstant: 1 > 44 <8D 01 00> storeIntoTemp: 1 inVectorAt: 0 > 47 <7D> blockReturn > 48 <7C> returnTop > > The closure [smallObject := 1] now, even though not referencing the largeObject, references > the tempVector. This tempVector holds on the to large object --> memory is wasted that is not needed. > > Back when implementing the SemChecker, I was thinking about this and wonderingâ¦ for sure > this can be optimized by allocating multiple arrays per scope, depending on how they are used later. > But even then I am sure you can construct strange cases (multiple nested blocks). > > And of course allocating 2 Arrays (or more) wastes memory, too. > > So this is not a simple problemâ¦ the question is how much it matters in practice, though. > > Marcus Thanks for the explanation. I am also not sure if this would be a real problem in practice. I guess most code can be rewritten to avoid it. But it is probably a good idea to document this, maybe in a unit test. And who knows, maybe one day we'll have to revisit this. Sven

June 29, 2013

Re: [Pharo-dev] Question about closure compilation

by Marcus Denker

>> > > OK, that seems OK. > > The reason I was asking was because I wanted to know if this JavaScript bug would hit us as well: > > http://point.davidglasser.net/2013/06/27/surprising-javascript-memory-leak.… > > It is slightly more complicated than the basic question of closing over an unreferenced variable though. > > I just can't figure out a way to quickly test it in Pharo. I think it is worth looking into. Yes, we have exactly the same problem. imagine a method like this: test | bigObject smallObject | [bigObject := Array new: 1000000] value. ^[smallObject := 1]. both big and small are escaping variables that are written. This means they are allocated in an Array on the heap (Cog is calling this a TempVector). There is just one tempVector created per scope, so we will have a tempVector with both smallObject and bigObject inside created for the method. This then is passed to the blocks, which access the temps via the vector. The bytecode looks like this: 21 <8A 02> push: (Array new: 2) 23 <68> popIntoTemp: 0 24 <10> pushTemp: 0 25 <8F 10 00 07> closureNumCopied: 1 numArgs: 0 bytes 29 to 35 29 <40> pushLit: Array 30 <21> pushConstant: 1000000 31 <CD> send: new: 32 <8D 00 00> storeIntoTemp: 0 inVectorAt: 0 35 <7D> blockReturn 36 <C9> send: value 37 <87> pop 38 <10> pushTemp: 0 39 <8F 10 00 05> closureNumCopied: 1 numArgs: 0 bytes 43 to 47 43 <76> pushConstant: 1 44 <8D 01 00> storeIntoTemp: 1 inVectorAt: 0 47 <7D> blockReturn 48 <7C> returnTop The closure [smallObject := 1] now, even though not referencing the largeObject, references the tempVector. This tempVector holds on the to large object --> memory is wasted that is not needed. Back when implementing the SemChecker, I was thinking about this and wonderingâ¦ for sure this can be optimized by allocating multiple arrays per scope, depending on how they are used later. But even then I am sure you can construct strange cases (multiple nested blocks). And of course allocating 2 Arrays (or more) wastes memory, too. So this is not a simple problemâ¦ the question is how much it matters in practice, though. Marcus

June 29, 2013