[Pharo-project] Big Image Tests
Today I tried to find out if it would be possible to allocate a large amount of memory in a VM. I used the official 32-bit VM (http://gforge.inria.fr/frs/download.php/29042/CogVM-Unix-13307.zip) and a Pharo 1.3 (http://gforge.inria.fr/frs/download.php/29273/Pharo-1.3-13315.zip) derived image on Ubuntu 11.04 (x86_64) Amazon AWS EC2 m1.large instance (about 8Gb RAM) and the following bash and Smalltalk scripts: run.sh #!/bin/bash script_home=$(dirname $0) script_home=$(cd $script_home && pwd) image=$script_home/pharo-t3.image script=$script_home/run.st args=$* vm=$script_home/bin/CogVM options="-mmap 1536m -vm-display-null -vm-sound-null" #echo $vm $options $image $script $args $vm $options $image $script $args run.st | args size monolithic object | FileStream stdout wantsLineEndConversion: true. Smalltalk garbageCollect. FileStream stdout nextPutAll: 'Running run.st with args: '; nextPutAll: Smalltalk commandLine arguments printString; cr; nextPutAll: Smalltalk vmStatisticsReportString. args := Smalltalk commandLine arguments. size := args first asInteger. monolithic := (args size > 1 and: [ args second = 'monolithic' ]). monolithic ifTrue: [ object := ByteArray ofSize: size * 1024 * 1024 ] ifFalse: [ object := Array ofSize: size * 1024. 1 to: size * 1024 do: [ :each | object at: each put: (ByteArray ofSize: 1024) ] ]. Smalltalk garbageCollect. FileStream stdout nextPutAll: 'Allocation of a size '; print: object size; space; print: object class; nextPutAll: ' OK'; cr; nextPutAll: Smalltalk vmStatisticsReportString. Smalltalk quitPrimitive. These were the results trying to allocate 1GB: $ ./run.sh 1024 Running run.st with args: #('1024') uptime 0h0m0s memory 49,103,224 bytes old 36,625,732 bytes (74.60000000000001%) young 9,812 bytes (0.0%) used 36,635,544 bytes (74.60000000000001%) free 12,467,680 bytes (25.400000000000002%) GCs 8 (90ms between GCs) full 2 totalling 212ms (29.400000000000002% uptime), avg 106.0ms incr 6 totalling 10ms (1.4000000000000001% uptime), avg 1.7000000000000002ms tenures 0 Allocation of a size 1048576 Array OK uptime 0h0m26s memory -1,007,414,928 bytes old -1,020,337,428 bytes (101.30000000000001%) young 5,888 bytes (0.0%) used -1,020,331,540 bytes (101.30000000000001%) free 12,916,612 bytes (-1.3%) GCs 590 (44ms between GCs) full 62 totalling 14,735ms (56.6% uptime), avg 237.70000000000002ms incr 528 totalling 10,225ms (39.300000000000004% uptime), avg 19.400000000000002ms tenures 519 (avg 1 GCs/tenure) Since last view 582 (44ms between GCs) uptime 25.3s full 60 totalling 14,523ms (57.400000000000006% uptime), avg 242.10000000000002ms incr 522 totalling 10,215ms (40.300000000000004% uptime), avg 19.6ms tenures 519 (avg 1 GCs/tenure) $ ./run.sh 1024 monolithic Running run.st with args: #('1024' 'monolithic') uptime 0h0m0s memory 49,103,216 bytes old 36,627,976 bytes (74.60000000000001%) young 9,844 bytes (0.0%) used 36,637,820 bytes (74.60000000000001%) free 12,465,396 bytes (25.400000000000002%) GCs 8 (46ms between GCs) full 2 totalling 208ms (56.1% uptime), avg 104.0ms incr 6 totalling 10ms (2.7% uptime), avg 1.7000000000000002ms tenures 0 Allocation of a size -1073741824 ByteArray OK uptime 0h0m2s memory -1,020,427,928 bytes old -1,037,112,380 bytes (101.60000000000001%) young 5,952 bytes (0.0%) used -1,037,106,428 bytes (101.60000000000001%) free 16,678,500 bytes (-1.6%) GCs 13 (177ms between GCs) full 4 totalling 725ms (31.5% uptime), avg 181.3ms incr 9 totalling 14ms (0.6000000000000001% uptime), avg 1.6ms tenures 0 Since last view 5 (386ms between GCs) uptime 1.9000000000000001s full 2 totalling 517ms (26.8% uptime), avg 258.5ms incr 3 totalling 4ms (0.2% uptime), avg 1.3ms tenures 0 Basically it seems to work. What I find particulary strange is that #vmStatisticsReportString starts reporting negative sizes, also the size of the 1GB ByteArray is reported as negative. Why ? Tomorrow I might try to find out how much further I can take this ? How close to 4GB ? Anybody else tried this ? Sven
On Tue, Dec 13, 2011 at 12:07:59AM +0100, Sven Van Caekenberghe wrote:
Today I tried to find out if it would be possible to allocate a large amount of memory in a VM.
<snip>
Basically it seems to work.
What I find particulary strange is that #vmStatisticsReportString starts reporting negative sizes, also the size of the 1GB ByteArray is reported as negative. Why ?
Tomorrow I might try to find out how much further I can take this ? How close to 4GB ?
Anybody else tried this ?
Hi Sven, At one time it was possible to get an image up to about 7.5 GB using a 64 bit image and VM, see attached picture. Links and information at <http://squeakvm.org/squeak64> Unfortunately VMMaker has regressed a bit in this regard and I can no longer repeat the experiment. When you see a big integer reported as negative, it is usually because of integer overflow, or from stuffing a large int value into a 32 bit int. Usually this is an issue with variable declarations, e.g. declaring a variable as signed when it is supposed to contain a positive value that may exceed the range of a signed int. Dave
On 13.12.2011 00:07, Sven Van Caekenberghe wrote:
Today I tried to find out if it would be possible to allocate a large amount of memory in a VM.
I used the official 32-bit VM (http://gforge.inria.fr/frs/download.php/29042/CogVM-Unix-13307.zip) and a Pharo 1.3 (http://gforge.inria.fr/frs/download.php/29273/Pharo-1.3-13315.zip) derived image on Ubuntu 11.04 (x86_64) Amazon AWS EC2 m1.large instance (about 8Gb RAM) and the following bash and Smalltalk scripts:
run.sh
#!/bin/bash script_home=$(dirname $0) script_home=$(cd $script_home&& pwd) image=$script_home/pharo-t3.image script=$script_home/run.st args=$* vm=$script_home/bin/CogVM options="-mmap 1536m -vm-display-null -vm-sound-null" #echo $vm $options $image $script $args $vm $options $image $script $args
run.st
| args size monolithic object |
FileStream stdout wantsLineEndConversion: true.
Smalltalk garbageCollect.
FileStream stdout nextPutAll: 'Running run.st with args: '; nextPutAll: Smalltalk commandLine arguments printString; cr; nextPutAll: Smalltalk vmStatisticsReportString.
args := Smalltalk commandLine arguments. size := args first asInteger. monolithic := (args size> 1 and: [ args second = 'monolithic' ]).
monolithic ifTrue: [ object := ByteArray ofSize: size * 1024 * 1024 ] ifFalse: [ object := Array ofSize: size * 1024. 1 to: size * 1024 do: [ :each | object at: each put: (ByteArray ofSize: 1024) ] ].
Smalltalk garbageCollect.
FileStream stdout nextPutAll: 'Allocation of a size '; print: object size; space; print: object class; nextPutAll: ' OK'; cr; nextPutAll: Smalltalk vmStatisticsReportString.
Smalltalk quitPrimitive.
These were the results trying to allocate 1GB:
$ ./run.sh 1024 Running run.st with args: #('1024') uptime 0h0m0s memory 49,103,224 bytes old 36,625,732 bytes (74.60000000000001%) young 9,812 bytes (0.0%) used 36,635,544 bytes (74.60000000000001%) free 12,467,680 bytes (25.400000000000002%) GCs 8 (90ms between GCs) full 2 totalling 212ms (29.400000000000002% uptime), avg 106.0ms incr 6 totalling 10ms (1.4000000000000001% uptime), avg 1.7000000000000002ms tenures 0 Allocation of a size 1048576 Array OK uptime 0h0m26s memory -1,007,414,928 bytes old -1,020,337,428 bytes (101.30000000000001%) young 5,888 bytes (0.0%) used -1,020,331,540 bytes (101.30000000000001%) free 12,916,612 bytes (-1.3%) GCs 590 (44ms between GCs) full 62 totalling 14,735ms (56.6% uptime), avg 237.70000000000002ms incr 528 totalling 10,225ms (39.300000000000004% uptime), avg 19.400000000000002ms tenures 519 (avg 1 GCs/tenure) Since last view 582 (44ms between GCs) uptime 25.3s full 60 totalling 14,523ms (57.400000000000006% uptime), avg 242.10000000000002ms incr 522 totalling 10,215ms (40.300000000000004% uptime), avg 19.6ms tenures 519 (avg 1 GCs/tenure)
$ ./run.sh 1024 monolithic Running run.st with args: #('1024' 'monolithic') uptime 0h0m0s memory 49,103,216 bytes old 36,627,976 bytes (74.60000000000001%) young 9,844 bytes (0.0%) used 36,637,820 bytes (74.60000000000001%) free 12,465,396 bytes (25.400000000000002%) GCs 8 (46ms between GCs) full 2 totalling 208ms (56.1% uptime), avg 104.0ms incr 6 totalling 10ms (2.7% uptime), avg 1.7000000000000002ms tenures 0 Allocation of a size -1073741824 ByteArray OK uptime 0h0m2s memory -1,020,427,928 bytes old -1,037,112,380 bytes (101.60000000000001%) young 5,952 bytes (0.0%) used -1,037,106,428 bytes (101.60000000000001%) free 16,678,500 bytes (-1.6%) GCs 13 (177ms between GCs) full 4 totalling 725ms (31.5% uptime), avg 181.3ms incr 9 totalling 14ms (0.6000000000000001% uptime), avg 1.6ms tenures 0 Since last view 5 (386ms between GCs) uptime 1.9000000000000001s full 2 totalling 517ms (26.8% uptime), avg 258.5ms incr 3 totalling 4ms (0.2% uptime), avg 1.3ms tenures 0
Basically it seems to work.
What I find particulary strange is that #vmStatisticsReportString starts reporting negative sizes, also the size of the 1GB ByteArray is reported as negative. Why ?
Tomorrow I might try to find out how much further I can take this ? How close to 4GB ?
Anybody else tried this ?
Sven
IIRC, all pointers are sqInts, not uints, so even if compiling with large-address-aware I doubt you will get a 32bit image above 2GB without seeing sporadic failures. Cheers, Henry
Hendrik, On 13 Dec 2011, at 12:00, Henrik Sperre Johansen wrote:
IIRC, all pointers are sqInts, not uints, so even if compiling with large-address-aware I doubt you will get a 32bit image above 2GB without seeing sporadic failures.
Yes, that is what I am seeing: I cannot raise -mmap above 2GB. Does that explain the negative sizes in #vmStatisticsReportString or for a 1GB ByteArray ? Sven
participants (3)
-
David T. Lewis -
Henrik Sperre Johansen -
Sven Van Caekenberghe