Way to embed efficiently data?
Hello, I want to know if there is an efficient way to embed data with version control on it(Monticello). By efficient, I mean without gzipping and Base64 encoding the data in the source code of a method. Ideally, I would like this embedded data in a separate file, to avoid memory consumption when I am not using this data. I am interesting on embedding text files(GLSL Shaders, OpenCL kernel source code) and binary data (images) Greetings, Ronie
The way I do it is by using a separate version control (GIT) for the external files and I then ask the jenkins to build me a repository including the image with my code from monticello and the files from GIT. To commit you need to use either monticello or git depending on what you edit. There might be a better solution though ... 2014-10-26 21:22 GMT+01:00 Ronie Salgado <roniesalg@gmail.com>:
Hello,
I want to know if there is an efficient way to embed data with version control on it(Monticello).
By efficient, I mean without gzipping and Base64 encoding the data in the source code of a method.
Ideally, I would like this embedded data in a separate file, to avoid memory consumption when I am not using this data.
I am interesting on embedding text files(GLSL Shaders, OpenCL kernel source code) and binary data (images)
Greetings, Ronie
On 26/10/14 15:22, Ronie Salgado wrote:
Hello,
I want to know if there is an efficient way to embed data with version control on it(Monticello).
not yet.
By efficient, I mean without gzipping and Base64 encoding the data in the source code of a method.
Ideally, I would like this embedded data in a separate file, to avoid memory consumption when I am not using this data.
For now embedd it and yes we should work on that.
I am interesting on embedding text files(GLSL Shaders, OpenCL kernel source code) and binary data (images)
Greetings, Ronie
I donât think that storing binary files in git is good practice git shines on declarative content But Google about it and see how other people experiences resulted. Are they happy or not?
On Oct 26, 2014, at 6:22 PM, Ronie Salgado <roniesalg@gmail.com> wrote:
Hello,
I want to know if there is an efficient way to embed data with version control on it(Monticello).
By efficient, I mean without gzipping and Base64 encoding the data in the source code of a method.
Ideally, I would like this embedded data in a separate file, to avoid memory consumption when I am not using this data.
I am interesting on embedding text files(GLSL Shaders, OpenCL kernel source code) and binary data (images)
Greetings, Ronie
2014-10-26 20:14 GMT-03:00 Sebastian Sastre <sebastian@flowingconcept.com>:
I donât think that storing binary files in git is good practice
git shines on declarative content
But Google about it and see how other people experiences resulted. Are they happy or not?
It's not efficient in terms of storage, but I know of people who uses git to manage PSD files and other "non code" files. I'm particularly interested in git for Pharo for the same reason, I want to be able to commit everything (Pharo and static files) in a single command. Esteban A. Maringolo
On Oct 26, 2014, at 9:22 PM, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
2014-10-26 20:14 GMT-03:00 Sebastian Sastre <sebastian@flowingconcept.com>:
I donât think that storing binary files in git is good practice
git shines on declarative content
But Google about it and see how other people experiences resulted. Are they happy or not?
It's not efficient in terms of storage, but I know of people who uses git to manage PSD files and other "non code" files.
I'm particularly interested in git for Pharo for the same reason, I want to be able to commit everything (Pharo and static files) in a single command.
Well, then you should use that in the exact same way you would commit in one command entire VirtualBox guests VMs git is a tool suposed to help in integrating work and not make your life too hard when solving merge conflicts when they happen. If you do things that diverge from that then, well.. then you start eroding the profit itâs trying to give you
Esteban A. Maringolo
I've seen a couple configurations retrieve tar files from DropBox. http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity loads NaCl library (libsodium) from Dropbox. See ConfigurationOfNacl>>preLoad | url fileRef response | url := self platformLibraryUrl asZnUrl. fileRef := FileSystem disk workingDirectory / url pathSegments last. (response := ZnEasy get: url) isSuccess ifTrue: [ fileRef writeStreamDo: [ : stream | stream nextPutAll: response contents ] ] ifFalse: [ self error: 'Cannot download compiled library file' ]. self uncompress: fileRef. platformLibraryUrl " Answer a download URL for the compiled version of library " Smalltalk os isWin32 ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.zip' ]. Smalltalk os isUnix ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz' ]. Smalltalk os isMacOSX ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz' ]. self error: 'Cannot find library for your OS platform'. It looks like a good way to solve these things. Forget about Dropbox versioning. But with a proper naming scheme, it is doable. Phil On Mon, Oct 27, 2014 at 1:40 AM, Sebastian Sastre < sebastian@flowingconcept.com> wrote:
On Oct 26, 2014, at 9:22 PM, Esteban A. Maringolo <emaringolo@gmail.com> wrote:
2014-10-26 20:14 GMT-03:00 Sebastian Sastre < sebastian@flowingconcept.com>:
I donât think that storing binary files in git is good practice
git shines on declarative content
But Google about it and see how other people experiences resulted. Are they happy or not?
It's not efficient in terms of storage, but I know of people who uses git to manage PSD files and other "non code" files.
I'm particularly interested in git for Pharo for the same reason, I want to be able to commit everything (Pharo and static files) in a single command.
Well, then you should use that in the exact same way you would commit in one command entire VirtualBox guests VMs
git is a tool suposed to help in integrating work and not make your life too hard when solving merge conflicts when they happen. If you do things that diverge from that then, well.. then you start eroding the profit itâs trying to give you
Esteban A. Maringolo
Am 27.10.2014 um 02:01 schrieb phil@highoctane.be:
I've seen a couple configurations retrieve tar files from DropBox.
http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity <http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity> loads NaCl library (libsodium) from Dropbox.
See ConfigurationOfNacl>>preLoad
| url fileRef response |
url := self platformLibraryUrl asZnUrl. fileRef := FileSystem disk workingDirectory / url pathSegments last. (response := ZnEasy get: url) isSuccess ifTrue: [ fileRef writeStreamDo: [ : stream | stream nextPutAll: response contents ] ] ifFalse: [ self error: 'Cannot download compiled library file' ]. self uncompress: fileRef.
platformLibraryUrl " Answer a download URL for the compiled version of library "
Smalltalk os isWin32 ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.zip <http://dl.dropboxusercontent.com/u/103833630/libsodium.zip>' ]. Smalltalk os isUnix ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz <http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz>' ]. Smalltalk os isMacOSX ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz <http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz>' ]. self error: 'Cannot find library for your OS platform'.
It looks like a good way to solve these things. Forget about Dropbox versioning. But with a proper naming scheme, it is doable.
For me this sounds like a receipt for disaster in the long run. Version management is about to be able to reliably load versions. Moving parts outside of a controlled environment will make you unhappy sooner or later⦠Regards Andreas
Phil
On Mon, Oct 27, 2014 at 1:40 AM, Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>> wrote:
On Oct 26, 2014, at 9:22 PM, Esteban A. Maringolo <emaringolo@gmail.com <mailto:emaringolo@gmail.com>> wrote:
2014-10-26 20:14 GMT-03:00 Sebastian Sastre <sebastian@flowingconcept.com <mailto:sebastian@flowingconcept.com>>:
I donât think that storing binary files in git is good practice
git shines on declarative content
But Google about it and see how other people experiences resulted. Are they happy or not?
It's not efficient in terms of storage, but I know of people who uses git to manage PSD files and other "non code" files.
I'm particularly interested in git for Pharo for the same reason, I want to be able to commit everything (Pharo and static files) in a single command.
Well, then you should use that in the exact same way you would commit in one command entire VirtualBox guests VMs
git is a tool suposed to help in integrating work and not make your life too hard when solving merge conflicts when they happen. If you do things that diverge from that then, well.. then you start eroding the profit itâs trying to give you
Esteban A. Maringolo
2014-10-27 15:00 GMT-03:00 Andreas Wacknitz <a.wacknitz@gmx.de>:
Am 27.10.2014 um 02:01 schrieb phil@highoctane.be:
I've seen a couple configurations retrieve tar files from DropBox.
http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity loads NaCl library (libsodium) from Dropbox.
See ConfigurationOfNacl>>preLoad
| url fileRef response |
url := self platformLibraryUrl asZnUrl. fileRef := FileSystem disk workingDirectory / url pathSegments last. (response := ZnEasy get: url) isSuccess ifTrue: [ fileRef writeStreamDo: [ : stream | stream nextPutAll: response contents ] ] ifFalse: [ self error: 'Cannot download compiled library file' ]. self uncompress: fileRef.
platformLibraryUrl " Answer a download URL for the compiled version of library "
Smalltalk os isWin32 ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.zip' ]. Smalltalk os isUnix ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz' ]. Smalltalk os isMacOSX ifTrue: [ ^ ' http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz' ]. self error: 'Cannot find library for your OS platform'.
It looks like a good way to solve these things. Forget about Dropbox versioning. But with a proper naming scheme, it is doable.
For me this sounds like a receipt for disaster in the long run.
Is just a resource download, I am running an Iliad web site and two applications for over a year downloading resources like that and never failed, so no disasters.
Version management is about to be able to reliably load versions. Moving parts outside of a controlled environment will make you unhappy sooner or laterâ¦
As Phil said, that is not suitable for versioning, but there is nothing stopping you from implementing or adapting Monticello to the best version management you can imagine. Cheers, Hernán
On Mon, Oct 27, 2014 at 7:32 PM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
2014-10-27 15:00 GMT-03:00 Andreas Wacknitz <a.wacknitz@gmx.de>:
Am 27.10.2014 um 02:01 schrieb phil@highoctane.be:
I've seen a couple configurations retrieve tar files from DropBox.
http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity loads NaCl library (libsodium) from Dropbox.
See ConfigurationOfNacl>>preLoad
| url fileRef response |
url := self platformLibraryUrl asZnUrl. fileRef := FileSystem disk workingDirectory / url pathSegments last. (response := ZnEasy get: url) isSuccess ifTrue: [ fileRef writeStreamDo: [ : stream | stream nextPutAll: response contents ] ] ifFalse: [ self error: 'Cannot download compiled library file' ]. self uncompress: fileRef.
platformLibraryUrl " Answer a download URL for the compiled version of library "
Smalltalk os isWin32 ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.zip' ]. Smalltalk os isUnix ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz' ]. Smalltalk os isMacOSX ifTrue: [ ^ ' http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz' ]. self error: 'Cannot find library for your OS platform'.
It looks like a good way to solve these things. Forget about Dropbox versioning. But with a proper naming scheme, it is doable.
For me this sounds like a receipt for disaster in the long run.
Is just a resource download, I am running an Iliad web site and two applications for over a year downloading resources like that and never failed, so no disasters.
We may want to have some Pharo-binary-bucket somewhere on the web, like we do have smalltalkhub. A kind of binary gist. For this to be a reference, we need to have a community shared solution. What does exists? (S3?) For huge resources, bittorrent-style would be nice too. Phil
Version management is about to be able to reliably load versions. Moving parts outside of a controlled environment will make you unhappy sooner or laterâ¦
As Phil said, that is not suitable for versioning, but there is nothing stopping you from implementing or adapting Monticello to the best version management you can imagine.
Cheers,
Hernán
Could be nice... http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-personal-cloud-us... Phil On Mon, Oct 27, 2014 at 7:32 PM, Hernán Morales Durand < hernan.morales@gmail.com> wrote:
2014-10-27 15:00 GMT-03:00 Andreas Wacknitz <a.wacknitz@gmx.de>:
Am 27.10.2014 um 02:01 schrieb phil@highoctane.be:
I've seen a couple configurations retrieve tar files from DropBox.
http://www.smalltalkhub.com/#!/~hernan/ApplicationSecurity loads NaCl library (libsodium) from Dropbox.
See ConfigurationOfNacl>>preLoad
| url fileRef response |
url := self platformLibraryUrl asZnUrl. fileRef := FileSystem disk workingDirectory / url pathSegments last. (response := ZnEasy get: url) isSuccess ifTrue: [ fileRef writeStreamDo: [ : stream | stream nextPutAll: response contents ] ] ifFalse: [ self error: 'Cannot download compiled library file' ]. self uncompress: fileRef.
platformLibraryUrl " Answer a download URL for the compiled version of library "
Smalltalk os isWin32 ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.zip' ]. Smalltalk os isUnix ifTrue: [ ^ 'http://dl.dropboxusercontent.com/u/103833630/libsodium.so.gz' ]. Smalltalk os isMacOSX ifTrue: [ ^ ' http://dl.dropboxusercontent.com/u/103833630/libsodium.dylib.gz' ]. self error: 'Cannot find library for your OS platform'.
It looks like a good way to solve these things. Forget about Dropbox versioning. But with a proper naming scheme, it is doable.
For me this sounds like a receipt for disaster in the long run.
Is just a resource download, I am running an Iliad web site and two applications for over a year downloading resources like that and never failed, so no disasters.
Version management is about to be able to reliably load versions. Moving parts outside of a controlled environment will make you unhappy sooner or laterâ¦
As Phil said, that is not suitable for versioning, but there is nothing stopping you from implementing or adapting Monticello to the best version management you can imagine.
Cheers,
Hernán
philippeback wrote
Could be nice...
http://blog.bittorrent.com/2013/05/23/how-i-created-my-own-personal-cloud-us...
Phil
If you go that route don't install ownCloud via the repositories: https://lists.ubuntu.com/archives/ubuntu-devel/2014-October/038522.html Instead use their offical versions -- View this message in context: http://forum.world.st/Way-to-embed-efficiently-data-tp4786769p4787445.html Sent from the Pharo Smalltalk Developers mailing list archive at Nabble.com.
Hello Ronie, iStoa, an interactive math exercices books (http://istoa.drgeo.eu), comes with a lot of binary files (PNG files). To manage that, I use Smalltalk hub to host Smalltalk code, and Launchpad hosting (bazar versioning system) to host the PNG files, the SVG counterpart source of these binary files, the related user documentations, the translation files, the Bash and Smalltalk build scripts, the VM to build the end user application and any other needed data. As with any application written in other programming language, the data are loaded at runtime. Btw, a LRU cache is still needed for iStoa, but it will came later. I suggest you keep your data out of Monticello, it will be just too complicate to manage your data, particularly if it grows or need to be updated. Good work! Hilaire Le 26/10/2014 21:22, Ronie Salgado a écrit :
Hello,
I want to know if there is an efficient way to embed data with version control on it(Monticello).
By efficient, I mean without gzipping and Base64 encoding the data in the source code of a method.
Ideally, I would like this embedded data in a separate file, to avoid memory consumption when I am not using this data.
I am interesting on embedding text files(GLSL Shaders, OpenCL kernel source code) and binary data (images)
Greetings, Ronie
-- Dr. Geo - http://drgeo.eu iStoa - http://istoa.drgeo.eu
participants (10)
-
Andreas Wacknitz -
Clément Bera -
Esteban A. Maringolo -
Hernán Morales Durand -
Hilaire -
Paul DeBruicker -
phil＠highoctane.be -
Ronie Salgado -
Sebastian Sastre -
stepharo