- Pharo-dev - lists.pharo.org

Re: Array sum. is very slow

by stephane ducasse

Thanks benoit for the snippet I run it in Pharo 10 and I got Test with 10000000 elements Original #sum -> Time: 195 milliseconds, Total: 4.999452880735064e6 Naive #sum -> Time: 153 milliseconds, Total: 4.999452880735063e6 Inject #sum -> Time: 198 milliseconds, Total: 4.999452880735063e6 in Pharo 9 Test with 10000000 elements Original #sum -> Time: 182 milliseconds, Total: 4.999339450212771e6 Naive #sum -> Time: 148 milliseconds, Total: 4.999339450212771e6 Inject #sum -> Time: 203 milliseconds, Total: 4.999339450212771e6 Iâm interested to understand why Pharo is slower. May be this is the impact of the new full blocks. We started to play with the idea of regression benchmarks. S > On 7 Jan 2022, at 16:36, Benoit St-Jean via Pharo-dev <pharo-dev(a)lists.pharo.org> wrote: > > Can you come up with a simple "base case" so we can find the bottleneck/problem? > > I'm not sure about what you're trying to do. > > What do you get if you try this in a workspace (adjust the value of n to what you want, I tested it with 10 million items). > > Let's get this one step at a time! > > > > | floatArray n rng t1 t2 t3 r1 r2 r3 | > > n := 10000000. > > rng := Random new. > > floatArray := Array new: n. > floatArray doWithIndex: [:each :idx | floatArray at: idx put: rng next]. > > t1 := Time millisecondsToRun: [r1 := floatArray sum]. > t2 := Time millisecondsToRun: [| total | > > total := 0. > floatArray do: [:each | total := total + each ]. > r2 := total]. > > t3 := Time millisecondsToRun: [r3 := floatArray inject: 0 into: [: total :each | total + each ]]. > > Transcript cr. > Transcript cr; show: 'Test with ', n printString, ' elements'. > Transcript cr;show: 'Original #sum -> Time: ', t1 printString, ' milliseconds, Total: ', r1 printString. > Transcript cr;show: 'Naive #sum -> Time: ', t2 printString, ' milliseconds, Total: ', r2 printString. > Transcript cr;show: 'Inject #sum -> Time: ', t3 printString, ' milliseconds, Total: ', r3 printString. > > -------------------------- > > Here are the results I get on Squeak 5.3 > > Test with 10000000 elements > Original #sum -> Time: 143 milliseconds, Total: 4.999271889099622e6 > Naive #sum -> Time: 115 milliseconds, Total: 4.999271889099622e6 > Inject #sum -> Time: 102 milliseconds, Total: 4.999271889099622e6 > > > > ----------------- > BenoÃ®t St-Jean > Yahoo! Messenger: bstjean > Twitter: @BenLeChialeux > Pinterest: benoitstjean > Instagram: Chef_Benito > IRC: lamneth > GitHub: bstjean > Blogue: endormitoire.wordpress.com > "A standpoint is an intellectual horizon of radius zero". (A. Einstein) > > > On Thursday, January 6, 2022, 03:38:22 p.m. EST, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: > > > I have written a micro benchmark which stresses a language in areas > which are crucial to my application. > > I have written this micro benchmark in Pharo, Crystal, Nim, Python, > PicoLisp, C, C++, Java and Julia. > > On my i7 laptop Julia completes it in about 1 minute and 15 seconds, > amazing magic they have done. > > Crystal and Nim do it in about 5 minutes. Python in about 25 minutes. > Pharo takes over 2 hours. :( > > In my benchmarks if I comment out the sum and average of the array. It > completes in 3.5 seconds. > And when I sum the array it gives the correct results. So I can verify > its validity. > > To illustrate below is some sample code of what I am doing. I iterate > over the array and do calculations on each value of the array and update > the array and sum and average at each value simple to stress array > access and sum and average. > > 28800 is simply derived from time series one minute values for 5 days, 4 > weeks. > > randarray := Array new: 28800. > > 1 to: randarray size do: [ :i | randarray at: i put: Number random ]. > > randarrayttr := [ 1 to: randarray size do: [ :i | "other calculations > here." randarray sum. randarray average ]] timeToRun. > > randarrayttr. "0:00:00:36.135" > > > I do 2 loops with 100 iterations each. > > randarrayttr * 200. "0:02:00:27" > > > I learned early on in this adventure when dealing with compiled > languages that if you donât do a lot, the test may not last long enough > to give any times. > > Pharo is my preference. But this is an awful big gap in performance. > When doing backtesting this is huge. Does my backtest take minutes, > hours or days? > > I am not a computer scientist nor expert in Pharo or Smalltalk. So I do > not know if there is anything which can improve this. > > > However I have played around with several experiments of my #sum: method. > > This implementation reduces the time on the above randarray in half. > > sum: col > | sum | > sum := 0. > 1 to: col size do: [ :i | > sum := sum + (col at: i) ]. > ^ sum > > randarrayttr2 := [ 1 to: randarray size do: [ :i | "other calculations > here." > ltsa sum: randarray. ltsa sum: randarray ]] timeToRun. > randarrayttr2. "0:00:00:18.563" > > And this one reduces it a little more. > > sum10: col > | sum | > sum := 0. > 1 to: ((col size quo: 10) * 10) by: 10 do: [ :i | > sum := sum + (col at: i) + (col at: (i + 1)) + (col at: (i + 2)) + > (col at: (i + 3)) + (col at: (i + 4)) > + (col at: (i + 5)) + (col at: (i + 6)) + (col at: (i + 7)) + > (col at: (i + 8)) + (col at: (i + 9))]. > ((col size quo: 10) * 10 + 1) to: col size do: [ :i | > sum := sum + (col at: i)]. > ^ sum > > randarrayttr3 := [ 1 to: randarray size do: [ :i | "other calculations > here." > ltsa sum10: randarray. ltsa sum10: randarray ]] timeToRun. > randarrayttr3. "0:00:00:14.592" > > It closes the gap with plain Python3 no numpy. But that is a pretty low > standard. > > Any ideas, thoughts, wisdom, directions to pursue. > > Thanks > > Jimmie >

Jan. 8, 2022

Re: Array sum. is very slow

by Sven Van Caekenberghe

Hi Jimmy, I made a couple more changes: - I added SequenceableCollection>>#sum | sum | sum := 0. 1 to: self size do: [ :each | sum := sum + (self at: each) ]. ^ sum as an extension method. It is not 100% semantically the same as the original, but it works for our case here. this also optimises #average BTW. This is the main one. - I tried to avoid a couple of integer -> float conversions in normalize: n | nn | nn := n = 0 ifTrue: [ 0.000123456789 ] ifFalse: [ n asFloat ]. [ nn <= 0.0001 ] whileTrue: [ nn := nn * 10.0 ]. [ nn >= 1.0 ] whileTrue: [ nn := nn * 0.1 ]. ^ nn - Avoided one assignment in loop1calc: i j: j n: n | v | v := n * (i+n) * (j-n) * 0.1234567. ^ self normalize: (v*v*v) the time for 10 iterations now is halved: === Starting test for array size: 28800 iterations: 10 Creating array of size: 28800 timeToRun: 0:00:00:00.002 Starting loop 1 at: 2022-01-07T19:28:52.109011+01:00 Loop 1 time: nil nsum: 11234.235001659386 navg: 0.3900776042242842 Starting loop 2 at: 2022-01-07T19:31:21.821784+01:00 Loop 2 time: 0:00:02:28.017 nsum: 11245.697629561537 navg: 0.3904756121375534 End of test. TotalTime: 0:00:04:57.733 === Sven > On 7 Jan 2022, at 16:30, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > > >> On 7 Jan 2022, at 16:05, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> >> Hello Sven, >> >> I went and removed the Stdouts that you mention and other timing code from the loops. >> >> I am running the test now, to see if that makes much difference. I do not think it will. >> >> The reason I put that in there is because it take so long to run. It can be frustrating to wait and wait and not know if your test is doing anything or not. So I put the code in to let me know. >> >> One of your parameters is incorrect. It is 100 iterations not 10. > > Ah, I misread the Python code, on top it says, reps = 10, while at the bottom it does indeed say, doit(100). > > So the time should be multiplied by 10. > > The logging, esp. the #flush will slow things down. But the removing the message tally spy is important too. > > The general implementation of #sum is not optimal in the case of a fixed array. Consider: > > data := Array new: 1e5 withAll: 0.5. > > [ data sum ] bench. "'494.503 per second'" > > [ | sum | sum := 0. data do: [ :each | sum := sum + each ]. sum ] bench. "'680.128 per second'" > > [ | sum | sum := 0. 1 to: 1e5 do: [ :each | sum := sum + (data at: each) ]. sum ] bench. "'1033.180 per second'" > > As others have remarked: doing #average right after #sum is doing the same thing twice. But maybe that is not the point. > >> I learned early on in this experiment that I have to do a large number of iterations or C, C++, Java, etc are too fast to have comprehensible results. >> >> I can tell if any of the implementations is incorrect by the final nsum. All implementations must produce the same result. >> >> Thanks for the comments. >> >> Jimmie >> >> >> On 1/7/22 07:40, Sven Van Caekenberghe wrote: >>> Hi Jimmie, >>> >>> I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 >>> >>> I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) (not done in Python either) as well as the MessageTally spyOn: from #run (slows things down). >>> >>> Then I ran your code with: >>> >>> [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. >>> >>> which gave me "0:00:09:31.338" >>> >>> The console output was: >>> >>> === >>> Starting test for array size: 28800 iterations: 10 >>> >>> Creating array of size: 28800 timeToRun: 0:00:00:00.031 >>> >>> Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 >>> Loop 1 time: nil >>> nsum: 11234.235001659388 >>> navg: 0.39007760422428434 >>> >>> Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 >>> Loop 2 time: 0:00:04:44.593 >>> nsum: 11245.697629561537 >>> navg: 0.3904756121375534 >>> >>> End of test. TotalTime: 0:00:09:31.338 >>> === >>> >>> Which would be twice as fast as Python, if I got the parameters correct. >>> >>> Sven >>> >>>> On 7 Jan 2022, at 13:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>>> >>>> As I stated this is a micro benchmark and very much not anything resembling a real app, Your comments are true if you are writing your app. But if you want to stress the language you are going to do things which are seemingly non-sense and abusive. >>>> >>>> Also as I stated. The test has to be sufficient to stress faster languages or it is meaningless. >>>> >>>> If I remove the #sum and the #average calls from the inner loops, this is what we get. >>>> >>>> Julia 0.2256 seconds >>>> Python 5.318 seconds >>>> Pharo 3.5 seconds >>>> >>>> This test does not sufficiently stress the language. Nor does it provide any valuable insight into summing and averaging which is done a lot, in lots of places in every iteration. >>>> >>>> If you notice that inner array changes the array every iteration. So every call to #sum and #average is getting different data. >>>> >>>> Full Test >>>> >>>> Julia 1.13 minutes >>>> Python 24.02 minutes >>>> Pharo 2:09:04 >>>> >>>> Code for the above is now published. You can let me know if I am doing something unequal to the various languages. >>>> >>>> And just remember anything you do which sufficiently changes the test has to be done in all the languages to give a fair test. This isn't a lets make Pharo look good test. I do want Pharo to look good, but honestly. >>>> >>>> Yes, I know that I can bind to BLAS or other external libraries. But that is not a test of Pharo. The Python is plain Python3 no Numpy, just using the the default list [] for the array. >>>> >>>> Julia is a whole other world. It is faster than Numpy. This is their domain and they optimize, optimize, optimize all the math. In fact they have reached the point that some pure Julia code beats pure Fortran. >>>> >>>> In all of this I just want Pharo to do the best it can. >>>> >>>> With the above results unless you already had an investment in Pharo, you wouldn't even look. :( >>>> >>>> Thanks for exploring this with me. >>>> >>>> >>>> Jimmie >>>> >>>> >>>> >>>> >>>> On 1/6/22 18:24, John Brant wrote: >>>>> On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>>>>> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >>>>>> >>>>>> Number random. "generates a float 0.8188008774329387" >>>>>> >>>>>> So in the randarray below it is an array of 28800 floats. >>>>>> >>>>>> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >>>>>> >>>>>> >>>>>> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >>>>>> >>>>>> >>>>>> https://github.com/jlhouchin/LanguageTestPharo >>>>>> >>>>>> >>>>>> Let me know if there is anything else I can do to help solve this problem. >>>>>> >>>>>> I am a lone developer in my spare time. So my apologies for any ugly code. >>>>>> >>>>> Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: >>>>> >>>>> 1 to: nsize do: [ :j || n | >>>>> n := narray at: j. >>>>> narray at: j put: (self loop1calc: i j: j n: n). >>>>> nsum := narray sum. >>>>> navg := narray average ] >>>>> >>>>> As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. >>>>> >>>>> >>>>> John Brant

Jan. 7, 2022

Re: Array sum. is very slow

by Benoit St-Jean

Can you come up with a simple "base case" so we can find the bottleneck/problem? I'm not sure about what you're trying to do. What do you get if you try this in a workspace (adjust the value of n to what you want, I tested it with 10 million items). Let's get this one step at a time! |Â floatArrayÂ nÂ rng t1 t2 t3 r1 r2 r3 | n := 10000000. rng := Random new. floatArray := Array new: n.Â floatArray doWithIndex: [:each :idx | floatArray at: idx put: rng next]. t1 := Time millisecondsToRun: [r1 := floatArray sum].t2 := Time millisecondsToRun: [| total | total := 0. floatArray do: [:each | total := total + each ]. r2 := total]. t3 := Time millisecondsToRun: [r3 := floatArray inject: 0 into:Â [: total :each | total + each ]]. Transcript cr.Transcript cr; show: 'Test with ', n printString, ' elements'.Transcript cr;show: 'Original #sum -> Time: ', t1 printString, ' milliseconds, Total: ', r1 printString.Transcript cr;show: 'Naive #sum -> Time: ', t2 printString, ' milliseconds, Total: ', r2 printString.Â Â Transcript cr;show: 'Inject #sum -> Time: ', t3 printString, ' milliseconds, Total: ', r3 printString.Â Â -------------------------- Here are the results I get on Squeak 5.3 Test with 10000000 elementsOriginal #sum -> Time: 143 milliseconds, Total: 4.999271889099622e6Naive #sum -> Time: 115 milliseconds, Total: 4.999271889099622e6Inject #sum -> Time: 102 milliseconds, Total: 4.999271889099622e6 ----------------- BenoÃ®t St-Jean Yahoo! Messenger: bstjean Twitter: @BenLeChialeux Pinterest: benoitstjean Instagram: Chef_Benito IRC: lamneth GitHub: bstjean Blogue: endormitoire.wordpress.com "A standpoint is an intellectual horizon of radius zero".Â (A. Einstein) On Thursday, January 6, 2022, 03:38:22 p.m. EST, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: I have written a micro benchmark which stresses a language in areas which are crucial to my application. I have written this micro benchmark in Pharo, Crystal, Nim, Python, PicoLisp, C, C++, Java and Julia. On my i7 laptop Julia completes it in about 1 minute and 15 seconds, amazing magic they have done. Crystal and Nim do it in about 5 minutes. Python in about 25 minutes. Pharo takes over 2 hours. :( In my benchmarks if I comment out the sum and average of the array. It completes in 3.5 seconds. And when I sum the array it gives the correct results. So I can verify its validity. To illustrate below is some sample code of what I am doing. I iterate over the array and do calculations on each value of the array and update the array and sum and average at each value simple to stress array access and sum and average. 28800 is simply derived from time series one minute values for 5 days, 4 weeks. randarray := Array new: 28800. 1 to: randarray size do: [ :i | randarray at: i put: Number random ]. randarrayttr := [ 1 to: randarray size do: [ :i | "other calculations here." randarray sum. randarray average ]] timeToRun. randarrayttr. "0:00:00:36.135" I do 2 loops with 100 iterations each. randarrayttr * 200. "0:02:00:27" I learned early on in this adventure when dealing with compiled languages that if you donât do a lot, the test may not last long enough to give any times. Pharo is my preference. But this is an awful big gap in performance. When doing backtesting this is huge. Does my backtest take minutes, hours or days? I am not a computer scientist nor expert in Pharo or Smalltalk. So I do not know if there is anything which can improve this. However I have played around with several experiments of my #sum: method. This implementation reduces the time on the above randarray in half. sum: col | sum | sum := 0. 1 to: col size do: [ :i | Â Â Â Â sum := sum + (col at: i) ]. ^ sum randarrayttr2 := [ 1 to: randarray size do: [ :i | "other calculations here." Â Â Â ltsa sum: randarray. ltsa sum: randarray ]] timeToRun. randarrayttr2. "0:00:00:18.563" And this one reduces it a little more. sum10: col | sum | sum := 0. 1 to: ((col size quo: 10) * 10) by: 10 do: [ :i | Â Â Â Â sum := sum + (col at: i) + (col at: (i + 1)) + (col at: (i + 2)) + (col at: (i + 3)) + (col at: (i + 4)) Â Â Â Â Â Â Â Â + (col at: (i + 5)) + (col at: (i + 6)) + (col at: (i + 7)) + (col at: (i + 8)) + (col at: (i + 9))]. ((col size quo: 10) * 10 + 1) to: col size do: [ :i | Â Â Â Â sum := sum + (col at: i)]. ^ sum randarrayttr3 := [ 1 to: randarray size do: [ :i | "other calculations here." Â Â Â ltsa sum10: randarray. ltsa sum10: randarray ]] timeToRun. randarrayttr3. "0:00:00:14.592" It closes the gap with plain Python3 no numpy. But that is a pretty low standard. Any ideas, thoughts, wisdom, directions to pursue. Thanks Jimmie

Jan. 7, 2022

Re: Array sum. is very slow

by Sven Van Caekenberghe

> On 7 Jan 2022, at 16:05, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: > > Hello Sven, > > I went and removed the Stdouts that you mention and other timing code from the loops. > > I am running the test now, to see if that makes much difference. I do not think it will. > > The reason I put that in there is because it take so long to run. It can be frustrating to wait and wait and not know if your test is doing anything or not. So I put the code in to let me know. > > One of your parameters is incorrect. It is 100 iterations not 10. Ah, I misread the Python code, on top it says, reps = 10, while at the bottom it does indeed say, doit(100). So the time should be multiplied by 10. The logging, esp. the #flush will slow things down. But the removing the message tally spy is important too. The general implementation of #sum is not optimal in the case of a fixed array. Consider: data := Array new: 1e5 withAll: 0.5. [ data sum ] bench. "'494.503 per second'" [ | sum | sum := 0. data do: [ :each | sum := sum + each ]. sum ] bench. "'680.128 per second'" [ | sum | sum := 0. 1 to: 1e5 do: [ :each | sum := sum + (data at: each) ]. sum ] bench. "'1033.180 per second'" As others have remarked: doing #average right after #sum is doing the same thing twice. But maybe that is not the point. > I learned early on in this experiment that I have to do a large number of iterations or C, C++, Java, etc are too fast to have comprehensible results. > > I can tell if any of the implementations is incorrect by the final nsum. All implementations must produce the same result. > > Thanks for the comments. > > Jimmie > > > On 1/7/22 07:40, Sven Van Caekenberghe wrote: >> Hi Jimmie, >> >> I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 >> >> I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) (not done in Python either) as well as the MessageTally spyOn: from #run (slows things down). >> >> Then I ran your code with: >> >> [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. >> >> which gave me "0:00:09:31.338" >> >> The console output was: >> >> === >> Starting test for array size: 28800 iterations: 10 >> >> Creating array of size: 28800 timeToRun: 0:00:00:00.031 >> >> Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 >> Loop 1 time: nil >> nsum: 11234.235001659388 >> navg: 0.39007760422428434 >> >> Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 >> Loop 2 time: 0:00:04:44.593 >> nsum: 11245.697629561537 >> navg: 0.3904756121375534 >> >> End of test. TotalTime: 0:00:09:31.338 >> === >> >> Which would be twice as fast as Python, if I got the parameters correct. >> >> Sven >> >>> On 7 Jan 2022, at 13:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>> >>> As I stated this is a micro benchmark and very much not anything resembling a real app, Your comments are true if you are writing your app. But if you want to stress the language you are going to do things which are seemingly non-sense and abusive. >>> >>> Also as I stated. The test has to be sufficient to stress faster languages or it is meaningless. >>> >>> If I remove the #sum and the #average calls from the inner loops, this is what we get. >>> >>> Julia 0.2256 seconds >>> Python 5.318 seconds >>> Pharo 3.5 seconds >>> >>> This test does not sufficiently stress the language. Nor does it provide any valuable insight into summing and averaging which is done a lot, in lots of places in every iteration. >>> >>> If you notice that inner array changes the array every iteration. So every call to #sum and #average is getting different data. >>> >>> Full Test >>> >>> Julia 1.13 minutes >>> Python 24.02 minutes >>> Pharo 2:09:04 >>> >>> Code for the above is now published. You can let me know if I am doing something unequal to the various languages. >>> >>> And just remember anything you do which sufficiently changes the test has to be done in all the languages to give a fair test. This isn't a lets make Pharo look good test. I do want Pharo to look good, but honestly. >>> >>> Yes, I know that I can bind to BLAS or other external libraries. But that is not a test of Pharo. The Python is plain Python3 no Numpy, just using the the default list [] for the array. >>> >>> Julia is a whole other world. It is faster than Numpy. This is their domain and they optimize, optimize, optimize all the math. In fact they have reached the point that some pure Julia code beats pure Fortran. >>> >>> In all of this I just want Pharo to do the best it can. >>> >>> With the above results unless you already had an investment in Pharo, you wouldn't even look. :( >>> >>> Thanks for exploring this with me. >>> >>> >>> Jimmie >>> >>> >>> >>> >>> On 1/6/22 18:24, John Brant wrote: >>>> On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>>>> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >>>>> >>>>> Number random. "generates a float 0.8188008774329387" >>>>> >>>>> So in the randarray below it is an array of 28800 floats. >>>>> >>>>> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >>>>> >>>>> >>>>> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >>>>> >>>>> >>>>> https://github.com/jlhouchin/LanguageTestPharo >>>>> >>>>> >>>>> Let me know if there is anything else I can do to help solve this problem. >>>>> >>>>> I am a lone developer in my spare time. So my apologies for any ugly code. >>>>> >>>> Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: >>>> >>>> 1 to: nsize do: [ :j || n | >>>> n := narray at: j. >>>> narray at: j put: (self loop1calc: i j: j n: n). >>>> nsum := narray sum. >>>> navg := narray average ] >>>> >>>> As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. >>>> >>>> >>>> John Brant

Jan. 7, 2022

Re: Array sum. is very slow

by Jimmie Houchin

Hello Sven, I went and removed the Stdouts that you mention and other timing code from the loops. I am running the test now, to see if that makes much difference. I do not think it will. The reason I put that in there is because it take so long to run. It can be frustrating to wait and wait and not know if your test is doing anything or not. So I put the code in to let me know. One of your parameters is incorrect. It is 100 iterations not 10. I learned early on in this experiment that I have to do a large number of iterations or C, C++, Java, etc are too fast to have comprehensible results. I can tell if any of the implementations is incorrect by the final nsum. All implementations must produce the same result. Thanks for the comments. Jimmie On 1/7/22 07:40, Sven Van Caekenberghe wrote: > Hi Jimmie, > > I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 > > I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) (not done in Python either) as well as the MessageTally spyOn: from #run (slows things down). > > Then I ran your code with: > > [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. > > which gave me "0:00:09:31.338" > > The console output was: > > === > Starting test for array size: 28800 iterations: 10 > > Creating array of size: 28800 timeToRun: 0:00:00:00.031 > > Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 > Loop 1 time: nil > nsum: 11234.235001659388 > navg: 0.39007760422428434 > > Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 > Loop 2 time: 0:00:04:44.593 > nsum: 11245.697629561537 > navg: 0.3904756121375534 > > End of test. TotalTime: 0:00:09:31.338 > === > > Which would be twice as fast as Python, if I got the parameters correct. > > Sven > >> On 7 Jan 2022, at 13:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> >> As I stated this is a micro benchmark and very much not anything resembling a real app, Your comments are true if you are writing your app. But if you want to stress the language you are going to do things which are seemingly non-sense and abusive. >> >> Also as I stated. The test has to be sufficient to stress faster languages or it is meaningless. >> >> If I remove the #sum and the #average calls from the inner loops, this is what we get. >> >> Julia 0.2256 seconds >> Python 5.318 seconds >> Pharo 3.5 seconds >> >> This test does not sufficiently stress the language. Nor does it provide any valuable insight into summing and averaging which is done a lot, in lots of places in every iteration. >> >> If you notice that inner array changes the array every iteration. So every call to #sum and #average is getting different data. >> >> Full Test >> >> Julia 1.13 minutes >> Python 24.02 minutes >> Pharo 2:09:04 >> >> Code for the above is now published. You can let me know if I am doing something unequal to the various languages. >> >> And just remember anything you do which sufficiently changes the test has to be done in all the languages to give a fair test. This isn't a lets make Pharo look good test. I do want Pharo to look good, but honestly. >> >> Yes, I know that I can bind to BLAS or other external libraries. But that is not a test of Pharo. The Python is plain Python3 no Numpy, just using the the default list [] for the array. >> >> Julia is a whole other world. It is faster than Numpy. This is their domain and they optimize, optimize, optimize all the math. In fact they have reached the point that some pure Julia code beats pure Fortran. >> >> In all of this I just want Pharo to do the best it can. >> >> With the above results unless you already had an investment in Pharo, you wouldn't even look. :( >> >> Thanks for exploring this with me. >> >> >> Jimmie >> >> >> >> >> On 1/6/22 18:24, John Brant wrote: >>> On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>>> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >>>> >>>> Number random. "generates a float 0.8188008774329387" >>>> >>>> So in the randarray below it is an array of 28800 floats. >>>> >>>> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >>>> >>>> >>>> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >>>> >>>> >>>> https://github.com/jlhouchin/LanguageTestPharo >>>> >>>> >>>> Let me know if there is anything else I can do to help solve this problem. >>>> >>>> I am a lone developer in my spare time. So my apologies for any ugly code. >>>> >>> Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: >>> >>> 1 to: nsize do: [ :j || n | >>> n := narray at: j. >>> narray at: j put: (self loop1calc: i j: j n: n). >>> nsum := narray sum. >>> navg := narray average ] >>> >>> As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. >>> >>> >>> John Brant

Jan. 7, 2022

Re: Array sum. is very slow

by Sven Van Caekenberghe

Hi Jimmie, I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) (not done in Python either) as well as the MessageTally spyOn: from #run (slows things down). Then I ran your code with: [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. which gave me "0:00:09:31.338" The console output was: === Starting test for array size: 28800 iterations: 10 Creating array of size: 28800 timeToRun: 0:00:00:00.031 Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 Loop 1 time: nil nsum: 11234.235001659388 navg: 0.39007760422428434 Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 Loop 2 time: 0:00:04:44.593 nsum: 11245.697629561537 navg: 0.3904756121375534 End of test. TotalTime: 0:00:09:31.338 === Which would be twice as fast as Python, if I got the parameters correct. Sven > On 7 Jan 2022, at 13:19, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: > > As I stated this is a micro benchmark and very much not anything resembling a real app, Your comments are true if you are writing your app. But if you want to stress the language you are going to do things which are seemingly non-sense and abusive. > > Also as I stated. The test has to be sufficient to stress faster languages or it is meaningless. > > If I remove the #sum and the #average calls from the inner loops, this is what we get. > > Julia 0.2256 seconds > Python 5.318 seconds > Pharo 3.5 seconds > > This test does not sufficiently stress the language. Nor does it provide any valuable insight into summing and averaging which is done a lot, in lots of places in every iteration. > > If you notice that inner array changes the array every iteration. So every call to #sum and #average is getting different data. > > Full Test > > Julia 1.13 minutes > Python 24.02 minutes > Pharo 2:09:04 > > Code for the above is now published. You can let me know if I am doing something unequal to the various languages. > > And just remember anything you do which sufficiently changes the test has to be done in all the languages to give a fair test. This isn't a lets make Pharo look good test. I do want Pharo to look good, but honestly. > > Yes, I know that I can bind to BLAS or other external libraries. But that is not a test of Pharo. The Python is plain Python3 no Numpy, just using the the default list [] for the array. > > Julia is a whole other world. It is faster than Numpy. This is their domain and they optimize, optimize, optimize all the math. In fact they have reached the point that some pure Julia code beats pure Fortran. > > In all of this I just want Pharo to do the best it can. > > With the above results unless you already had an investment in Pharo, you wouldn't even look. :( > > Thanks for exploring this with me. > > > Jimmie > > > > > On 1/6/22 18:24, John Brant wrote: >> On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >>> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >>> >>> Number random. "generates a float 0.8188008774329387" >>> >>> So in the randarray below it is an array of 28800 floats. >>> >>> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >>> >>> >>> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >>> >>> >>> https://github.com/jlhouchin/LanguageTestPharo >>> >>> >>> Let me know if there is anything else I can do to help solve this problem. >>> >>> I am a lone developer in my spare time. So my apologies for any ugly code. >>> >> Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: >> >> 1 to: nsize do: [ :j || n | >> n := narray at: j. >> narray at: j put: (self loop1calc: i j: j n: n). >> nsum := narray sum. >> navg := narray average ] >> >> As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. >> >> >> John Brant

Jan. 7, 2022

Re: Array sum. is very slow

by Jimmie Houchin

As I stated this is a micro benchmark and very much not anything resembling a real app, Your comments are true if you are writing your app. But if you want to stress the language you are going to do things which are seemingly non-sense and abusive. Also as I stated. The test has to be sufficient to stress faster languages or it is meaningless. If I remove the #sum and the #average calls from the inner loops, this is what we get. JuliaÂ Â Â 0.2256 seconds PythonÂ Â 5.318Â seconds PharoÂ Â Â 3.5Â Â Â seconds This test does not sufficiently stress the language. Nor does it provide any valuable insight into summing and averaging which is done a lot, in lots of places in every iteration. If you notice that inner array changes the array every iteration. So every call to #sum and #average is getting different data. Full Test JuliaÂ Â Â Â 1.13Â minutes PythonÂ Â 24.02 minutes PharoÂ Â Â 2:09:04 Code for the above is now published. You can let me know if I am doing something unequal to the various languages. And just remember anything you do which sufficiently changes the test has to be done in all the languages to give a fair test. This isn't a lets make Pharo look good test. I do want Pharo to look good, but honestly. Yes, I know that I can bind to BLAS or other external libraries. But that is not a test of Pharo. The Python is plain Python3 no Numpy, just using the the default list [] for the array. Julia is a whole other world. It is faster than Numpy. This is their domain and they optimize, optimize, optimize all the math. In fact they have reached the point that some pure Julia code beats pure Fortran. In all of this I just want Pharo to do the best it can. With the above results unless you already had an investment in Pharo, you wouldn't even look. :( Thanks for exploring this with me. Jimmie On 1/6/22 18:24, John Brant wrote: > On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com> wrote: >> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >> >> Number random. "generates a float 0.8188008774329387" >> >> So in the randarray below it is an array of 28800 floats. >> >> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >> >> >> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >> >> >> https://github.com/jlhouchin/LanguageTestPharo >> >> >> Let me know if there is anything else I can do to help solve this problem. >> >> I am a lone developer in my spare time. So my apologies for any ugly code. >> > Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: > > 1 to: nsize do: [ :j || n | > n := narray at: j. > narray at: j put: (self loop1calc: i j: j n: n). > nsum := narray sum. > navg := narray average ] > > As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. > > > John Brant

Jan. 7, 2022

This week (01/2022) on the Pharo Issue Tracker

by Marcus Denker

After christmas holidays, work started again this week: 5 PRs merged, 8 issues fixed: FIXES ===== - Fixes #10263 Setting minimum size for browser window #10686 https://github.com/pharo-project/pharo/pull/10686 - 9697-FTExamplesexampleTree1-menu-item-Expand-All-freezes-image #10630 https://github.com/pharo-project/pharo/pull/10630 - Fixes pharo-spec/Spec#1219: Ticking list should scroll to the next ticked items instead of scrolling up #10685 https://github.com/pharo-project/pharo/pull/10685 Improvements ============ - Add TFProcessLocalWorker #10668 https://github.com/pharo-project/pharo/pull/10668 - add-SystemDictionary-loopUpVarDeclare #10683 https://github.com/pharo-project/pharo/pull/10683

Jan. 7, 2022

Re: Array sum. is very slow

by Guillermo Polito

Yes, I just saw also that I used an interval instead of an arrayâ¦ I need to sleep more ^^ Anyways, even with a 28k large array wether they are small integers or floats, I have "reasonable resultsâ (where reasonable = not taking hours, nor minutes but a couple of milliseconds :P) randarray := Array new: 28800 withAll: 0. [randarray sum] bench "'2059.176 per second'" randarray2 := Array new: 28800 withAll: 0.1234567. [randarray2 sum] bench "'1771.737 per secondâ" I join Johnâs request to see the Python codeâ¦ Is that possible? G > El 6 ene 2022, a las 23:35, Jimmie Houchin <jlhouchin(a)gmail.com> escribiÃ³: > > No, it is an array of floats. The only integers in the test are in the indexes of the loops. > > Number random. "generates a float 0.8188008774329387" > > So in the randarray below it is an array of 28800 floats. > > It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... > > > I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. > > > https://github.com/jlhouchin/LanguageTestPharo > > > Let me know if there is anything else I can do to help solve this problem. > > I am a lone developer in my spare time. So my apologies for any ugly code. > > > Thanks for your help. > > Jimmie > > > On 1/6/22 15:07, Guillermo Polito wrote: >> Hi Jummie, >> >> Is it possible that your program is computing a lot of **very** large integers? >> >> Iâm just trying the following with small numbers, and I donât see the issue. #sum executes on a 28k large collection around 20 million times per second on my old 2015 i5. >> >> a := (1 to: 28000). >> [a sum] bench "'20256552.490 per secondâ" >> >> If you could share with us more data, we could take a look. >> Now iâm curious. >> >> Thanks, >> G >> >>> El 6 ene 2022, a las 21:37, Jimmie Houchin <jlhouchin(a)gmail.com> escribiÃ³: >>> >>> I have written a micro benchmark which stresses a language in areas which are crucial to my application. >>> >>> I have written this micro benchmark in Pharo, Crystal, Nim, Python, PicoLisp, C, C++, Java and Julia. >>> >>> On my i7 laptop Julia completes it in about 1 minute and 15 seconds, amazing magic they have done. >>> >>> Crystal and Nim do it in about 5 minutes. Python in about 25 minutes. Pharo takes over 2 hours. :( >>> >>> In my benchmarks if I comment out the sum and average of the array. It completes in 3.5 seconds. >>> And when I sum the array it gives the correct results. So I can verify its validity. >>> >>> To illustrate below is some sample code of what I am doing. I iterate over the array and do calculations on each value of the array and update the array and sum and average at each value simple to stress array access and sum and average. >>> >>> 28800 is simply derived from time series one minute values for 5 days, 4 weeks. >>> >>> randarray := Array new: 28800. >>> >>> 1 to: randarray size do: [ :i | randarray at: i put: Number random ]. >>> >>> randarrayttr := [ 1 to: randarray size do: [ :i | "other calculations here." randarray sum. randarray average ]] timeToRun. >>> >>> randarrayttr. "0:00:00:36.135" >>> >>> >>> I do 2 loops with 100 iterations each. >>> >>> randarrayttr * 200. "0:02:00:27" >>> >>> >>> I learned early on in this adventure when dealing with compiled languages that if you donât do a lot, the test may not last long enough to give any times. >>> >>> Pharo is my preference. But this is an awful big gap in performance. When doing backtesting this is huge. Does my backtest take minutes, hours or days? >>> >>> I am not a computer scientist nor expert in Pharo or Smalltalk. So I do not know if there is anything which can improve this. >>> >>> >>> However I have played around with several experiments of my #sum: method. >>> >>> This implementation reduces the time on the above randarray in half. >>> >>> sum: col >>> | sum | >>> sum := 0. >>> 1 to: col size do: [ :i | >>> sum := sum + (col at: i) ]. >>> ^ sum >>> >>> randarrayttr2 := [ 1 to: randarray size do: [ :i | "other calculations here." >>> ltsa sum: randarray. ltsa sum: randarray ]] timeToRun. >>> randarrayttr2. "0:00:00:18.563" >>> >>> And this one reduces it a little more. >>> >>> sum10: col >>> | sum | >>> sum := 0. >>> 1 to: ((col size quo: 10) * 10) by: 10 do: [ :i | >>> sum := sum + (col at: i) + (col at: (i + 1)) + (col at: (i + 2)) + (col at: (i + 3)) + (col at: (i + 4)) >>> + (col at: (i + 5)) + (col at: (i + 6)) + (col at: (i + 7)) + (col at: (i + 8)) + (col at: (i + 9))]. >>> ((col size quo: 10) * 10 + 1) to: col size do: [ :i | >>> sum := sum + (col at: i)]. >>> ^ sum >>> >>> randarrayttr3 := [ 1 to: randarray size do: [ :i | "other calculations here." >>> ltsa sum10: randarray. ltsa sum10: randarray ]] timeToRun. >>> randarrayttr3. "0:00:00:14.592" >>> >>> It closes the gap with plain Python3 no numpy. But that is a pretty low standard. >>> >>> Any ideas, thoughts, wisdom, directions to pursue. >>> >>> Thanks >>> >>> Jimmie >>>

Jan. 7, 2022

Re: Array sum. is very slow

by stephane ducasse

Thanks John This was an important remark :) Another remark is that you can also call BLAS for heavy mathematical operations (this is what numpy is doing just calling large fortran library and I do not know for julia but it should be same). And this is easy to do in Pharo. https://thepharo.dev/2021/10/17/binding-an-external-library-into-pharo/ <https://thepharo.dev/2021/10/17/binding-an-external-library-into-pharo/> And now you can just define a lot more easily a new binding. S > On 7 Jan 2022, at 01:24, John Brant <brant(a)refactoryworkers.com> wrote: > > On Jan 6, 2022, at 4:35 PM, Jimmie Houchin <jlhouchin(a)gmail.com <mailto:jlhouchin@gmail.com>> wrote: >> >> No, it is an array of floats. The only integers in the test are in the indexes of the loops. >> >> Number random. "generates a float 0.8188008774329387" >> >> So in the randarray below it is an array of 28800 floats. >> >> It just felt so wrong to me that Python3 was so much faster. I don't care if Nim, Crystal, Julia are faster. But... >> >> >> I am new to Iceberg and have never shared anything on Github so this is all new to me. I uploaded my language test so you can see what it does. It is a micro-benchmark. It does things that are not realistic in an app. But it does stress a language in areas important to my app. >> >> >> https://github.com/jlhouchin/LanguageTestPharo >> >> >> Let me know if there is anything else I can do to help solve this problem. >> >> I am a lone developer in my spare time. So my apologies for any ugly code. >> > > Are you sure that you have the same algorithm in Python? You are calling sum and average inside the loop where you are modifying the array: > > 1 to: nsize do: [ :j || n | > n := narray at: j. > narray at: j put: (self loop1calc: i j: j n: n). > nsum := narray sum. > navg := narray average ] > > As a result, you are calculating the sum of the 28,800 size array 28,800 times (plus another 28,800 times for the average). If I write a similar loop in Python, it looks like it would take almost 9 minutes on my machine without using numpy to calculate the sum. The Pharo code takes ~40 seconds. If this is really how the code should be, then I would change it to not call sum twice (once for sum and once in average). This will almost result in a 2x speedup. You could also modify the algorithm to update the nsum value in the loop instead of summing the array each time. I think the updating would require <120,000 math ops vs the >1.6 billion that you are performing. > > > John Brant

Jan. 7, 2022