On Sun, May 21, 2017 at 11:25 PM, Stephan Eggermont <stephan@stack.nl> wrote:

At the PharoDays I was painfully reminded that SSDs perform really badly when using small files. The Bloc tutorial used a github filetree repo and that has a lot of files. The whole folder is 116 MB in 16K files. Copying that amount of data should not be noticable, taking about a third of a second. With it being in so many files, it took more than half a minute, or a hundred times as long.

That is too much overhead. How can we improve the file format in a way that keeps the cross-platform exchange advantages and a reasonable way to view diffs and propose small changes using the github web tools?

Cuis uses a different format with git. How does that compare? What is used in Squeak?

So this was copying files at the OS level?�� Which OS?

For comparison, pulling whole of pharo-core across network and unpacking...
$ time git clone --depth=1��https://github.com/pharo-project/pharo-core.git

real 0m44.306s

user 0m5.384s

sys 0m5.908s

Count and size of files

$ ��find pharo-core/ | wc -l

152547 �� [files]

$ du -sh pharo-core
639MB

The smaller size of the archive without compression was a surprise. i.e. two-thirds empty space in filesystem blocks...

$ time tar cf pharo-core.tar pharo-core/

real 0m8.216s

user 0m2.284s

sys 0m3.776s

$ ls -lh pharo-core.tar��

212MB ��

File by file copy to a USB stick. Wow! Yes, thats over an hour.��

$ time cp -R ��pharo-core ��/media/ben/USBSTICK/pharo-core

real 68m49.168s ��

user 0m1.852s

sys 0m39.768s

Many forums indicated the following should be faster, but it was not...

$ time ((tar cf ��- ��pharo-core) | (tar xf ��- �� -C /media/ben/USBSTICK/))

real 77m47.896s

user 0m6.360s

sys 0m38.900s

Saving to an archive on USB stick is much quicker...

$ time tar cf ��/media/ben/USBSTICK/pharo-core.tar ��pharo-core

real 1m35.723s

user 0m3.188s

sys 0m5.972s

Saving to a compressed archive on USB stick is muchmuch quicker...

$ time tar zcf /media/ben/USBSTICK/pharo-core.tgz �� pharo-core

real 0m11.778s

user 0m7.908s

sys 0m3.604s

Saving to a compressed archive on USB stick, then unpacking that to local drive...

$ time (tar zcf /media/ben/USBSTICK/pharo-core.tgz �� pharo-core \

�� && tar zxf /media/ben/USBSTICK/pharo-core.tgz �� -C restore/)

real 0m49.934s

user 0m10.552s

sys 0m9.192s

So 38 seconds to unpack, which is very similar to original git clone.

But I guess your primary drive is SSD?�� So you don't get to bypass the issue using archvies?

What result do you get cloning pharo-core like my first test case?

cheers -ben��