Hi everyone, Over the last weeks I have started to explore IPFS more seriously. IPFS, the Inter-Planetary File System, is supposed to be the next-generation Web: a decentralized content-addressed database. Since there is nothing better then Pharo for exploring databases, I have started to write an IPFS interface to Pharo: https://github.com/khinsen/ipfs-pharo It connects to a local IPFS server, so you have to have one running. It's surprisingly straightforward to install and configure, unless you have to fight with firewalls that block IPFS traffic. Feedback of any kind is welcome! Cheers, Konrad.
On Thu, Jun 6, 2019 at 6:35 PM Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
Hi everyone,
Over the last weeks I have started to explore IPFS more seriously. IPFS, the Inter-Planetary File System, is supposed to be the next-generation Web: a decentralized content-addressed database.
Since there is nothing better then Pharo for exploring databases, I have started to write an IPFS interface to Pharo:
https://github.com/khinsen/ipfs-pharo
It connects to a local IPFS server, so you have to have one running. It's surprisingly straightforward to install and configure, unless you have to fight with firewalls that block IPFS traffic.
Great work ! I think Marcus was interested to have something like that. It is a solution to distribute scientific datasets ? A+ -- Serge Stinckwic h Int. Research Unit on Modelling/Simulation of Complex Systems (UMMISCO) Sorbonne University (SU) French National Research Institute for Sustainable Development (IRD) U niversity of Yaoundé I, Cameroon "Programs must be written for people to read, and only incidentally for machines to execute." https://twitter.com/SergeStinckwich
Am 06.06.19 um 19:37 schrieb Serge Stinckwich:
Great work ! I think Marcus was interested to have something like that. It is a solution to distribute scientific datasets ?
That's what I am trying to explore. But IPFS is a nice basis for many more applications. I have seen an IPFS-based GitHub (Radicle), various social networks, personal information management (Textile), and a lot more. Konrad.
On 6 Jun 2019, at 19:37, Serge Stinckwich <serge.stinckwich@gmail.com> wrote:
On Thu, Jun 6, 2019 at 6:35 PM Konrad Hinsen <konrad.hinsen@fastmail.net <mailto:konrad.hinsen@fastmail.net>> wrote: Hi everyone,
Over the last weeks I have started to explore IPFS more seriously. IPFS, the Inter-Planetary File System, is supposed to be the next-generation Web: a decentralized content-addressed database.
Since there is nothing better then Pharo for exploring databases, I have started to write an IPFS interface to Pharo:
https://github.com/khinsen/ipfs-pharo <https://github.com/khinsen/ipfs-pharo>
It connects to a local IPFS server, so you have to have one running. It's surprisingly straightforward to install and configure, unless you have to fight with firewalls that block IPFS traffic.
Great work ! I think Marcus was interested to have something like that.
yes! IPFS has nice properties for many things⦠e.g. one simple things to try is to distribute images via IPFS⦠it has all the propose of Bittorrent for that. Imagine your case in Africa⦠very nice for that already. Marcus
I would like something like that for Pharo images. Can you sync files.pharo.org on ipfs ? Sent from my iPhone
On 11 Jun 2019, at 17:00, Marcus Denker <marcus.denker@inria.fr> wrote:
On 6 Jun 2019, at 19:37, Serge Stinckwich <serge.stinckwich@gmail.com> wrote:
On Thu, Jun 6, 2019 at 6:35 PM Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
Hi everyone,
Over the last weeks I have started to explore IPFS more seriously. IPFS, the Inter-Planetary File System, is supposed to be the next-generation Web: a decentralized content-addressed database.
Since there is nothing better then Pharo for exploring databases, I have started to write an IPFS interface to Pharo:
https://github.com/khinsen/ipfs-pharo
It connects to a local IPFS server, so you have to have one running. It's surprisingly straightforward to install and configure, unless you have to fight with firewalls that block IPFS traffic.
Great work ! I think Marcus was interested to have something like that.
yes!
IPFS has nice properties for many things⦠e.g. one simple things to try is to distribute images via IPFS⦠it has all the propose of Bittorrent for that. Imagine your case in Africa⦠very nice for that already.
Marcus
On 11 Jun 2019, at 13:34, serge.stinckwich@gmail.com wrote:
I would like something like that for Pharo images. Can you sync files.pharo.org <http://files.pharo.org/> on ipfs ?
Yes, that would be the first step. We could even use this: https://blog.cloudflare.com/distributed-web-gateway/
Sent from my iPhone
On 11 Jun 2019, at 17:00, Marcus Denker <marcus.denker@inria.fr <mailto:marcus.denker@inria.fr>> wrote:
On 6 Jun 2019, at 19:37, Serge Stinckwich <serge.stinckwich@gmail.com <mailto:serge.stinckwich@gmail.com>> wrote:
On Thu, Jun 6, 2019 at 6:35 PM Konrad Hinsen <konrad.hinsen@fastmail.net <mailto:konrad.hinsen@fastmail.net>> wrote: Hi everyone,
Over the last weeks I have started to explore IPFS more seriously. IPFS, the Inter-Planetary File System, is supposed to be the next-generation Web: a decentralized content-addressed database.
Since there is nothing better then Pharo for exploring databases, I have started to write an IPFS interface to Pharo:
https://github.com/khinsen/ipfs-pharo <https://github.com/khinsen/ipfs-pharo>
It connects to a local IPFS server, so you have to have one running. It's surprisingly straightforward to install and configure, unless you have to fight with firewalls that block IPFS traffic.
Great work ! I think Marcus was interested to have something like that.
yes!
IPFS has nice properties for many things⦠e.g. one simple things to try is to distribute images via IPFS⦠it has all the propose of Bittorrent for that. Imagine your case in Africa⦠very nice for that already.
Marcus
serge.stinckwich@gmail.com writes:
I would like something like that for Pharo images. Can you sync files.pharo.org on ipfs ?
That should be straightforward using nothing but the IPFS command-line client. But it's worth thinking about the goals. What are you trying to improve exactly? The main advantage that IPFS provides is the possibility to create full or partial mirrors of the data (and thus faster access), including on your personal machine. To make this effective, you have to think a bit about good data management strategies. Few people will want to mirror all the history of Pharo. So you have to make interesting subcollections available under permanent names (via IPNS). Konrad.
On 11/06/2019 18:07, Stephan Eggermont wrote:
Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: :
. To make this effective, you have to think a bit about good data management strategies. Few people will want to mirror all the history of Pharo. Why? Fits on an SD card in a raspberry pi
All of http://files.pharo.org/ ? So how many GB is that? Konrad.
Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
On 11/06/2019 18:07, Stephan Eggermont wrote:
Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: :
. To make this effective, you have to think a bit about good data management strategies. Few people will want to mirror all the history of Pharo. Why? Fits on an SD card in a raspberry pi
All of http://files.pharo.org/ ? So how many GB is that?
It is only a few thousand changes per release. There is no reason why that shouldnât compress well Stephan
Stephan Eggermont <stephan@stack.nl> writes:
All of http://files.pharo.org/ ? So how many GB is that?
It is only a few thousand changes per release. There is no reason why that shouldnât compress well
Did anybody try? In IPFS, files are cut into blocks of roughly 256 KB. Blocks shared between multiple files are stored only once. So if changes in Pharo images from one version to the next happen mainly in one or two specific places (such as beginning and end), they would be stored efficiently on IPFS without any effort. But again, the best way to know is to try it out. Konrad.
On 13 Jun 2019, at 08:10, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
Stephan Eggermont <stephan@stack.nl> writes:
All of http://files.pharo.org/ ? So how many GB is that?
It is only a few thousand changes per release. There is no reason why that shouldnât compress well
Did anybody try?
In IPFS, files are cut into blocks of roughly 256 KB. Blocks shared between multiple files are stored only once. So if changes in Pharo images from one version to the next happen mainly in one or two specific places (such as beginning and end), they would be stored efficiently on IPFS without any effort.
But again, the best way to know is to try it out.
Indeed. Another thing to try: if someone has a local IPFS running, would it cache only those images that either this machine or near machines requested? This way we could have one âfullâ copy and caching (e.g. in the âAfrican class roomâ example) would happen automatically. Of course the downside is that one needs to speak the IPFS protocol, thus either running a client (e.g the go client)⦠so real transparent use would only be possible if Pharo could implement the protocol⦠(all based on a not complete understanding of IPFS) Marcus
On 13 Jun 2019, at 09:00, Marcus Denker <marcus.denker@inria.fr> wrote:
On 13 Jun 2019, at 08:10, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
Stephan Eggermont <stephan@stack.nl> writes:
All of http://files.pharo.org/ ? So how many GB is that?
It is only a few thousand changes per release. There is no reason why that shouldnât compress well
Did anybody try?
In IPFS, files are cut into blocks of roughly 256 KB. Blocks shared between multiple files are stored only once. So if changes in Pharo images from one version to the next happen mainly in one or two specific places (such as beginning and end), they would be stored efficiently on IPFS without any effort.
But again, the best way to know is to try it out.
Indeed. Another thing to try: if someone has a local IPFS running, would it cache only those images that either this machine or near machines requested?
This way we could have one âfullâ copy and caching (e.g. in the âAfrican class roomâ example) would happen automatically.
Of course the downside is that one needs to speak the IPFS protocol, thus either running a client (e.g the go client)⦠so real transparent use would only be possible if Pharo could implement the protocolâ¦
There are many other ideas to explore (based on my partial knowledge)⦠more brainstorming than anything else: - Could we have an image formate that would be âblock friendlyâ? - How could one move resources out of the image into IPFS? - Content addressing (like it is done in Git, too) and Objects, how can it mix? â Could one have âcontent pointersâ instead of âmemory pointersâ? I am sure there is much more⦠Marcus
Marcus Denker <marcus.denker@inria.fr> writes:
Indeed. Another thing to try: if someone has a local IPFS running, would it cache only those images that either this machine or near machines requested?
Yes. But note that "nearby" is probably not what you expect, since IPFS connects more or less randomly to other peers in the network. It's neither geographically nearby nor nearby in the local network. All that can be configured.
This way we could have one âfullâ copy and caching (e.g. in the âAfrican class roomâ example) would happen automatically.
That might work. A safer approach is to have one server machine in the classroom on which the required files are pinned (IPFS jargon for "not removed during garbage collection"). Then configure the other machines nearby to "bootstrap" from that server.
Of course the downside is that one needs to speak the IPFS protocol, thus either running a client (e.g the go client)⦠so real transparent use would only be possible if Pharo could implement the protocolâ¦
The generally recommended way to use IPFS on a desktop machine is what I do in my Pharo interface: connect via HTTP to go-ipfs running as a daemon process. Even the IPFS command-line client does that. The advantage is that you have only one data repository per machine, rather than one per process. For getting a Pharo image, you would then use the command-line interface, or implement IPFS access in Pharo launcher via the daemon (which should be straightforward, since it's really just an HTTP request).
There are many other ideas to explore (based on my partial knowledge)⦠more brainstorming than anything else:
- Could we have an image formate that would be âblock friendlyâ?
Probably. If some stable core is in one part of the image and the stuff that changes with each release in another part, then many block could be shared. Which IPFS does automatically.
- How could one move resources out of the image into IPFS?
This is what I am currently experimenting with. My idea is to use IPFS as something like an extended image, mainly for data that I would also like to access from non-Pharo code.
- Content addressing (like it is done in Git, too) and Objects, how can it mix? â Could one have âcontent pointersâ instead of âmemory pointersâ?
It can certainly be done (just put all objects into a dictionary with the hash as the key), but memory efficiency and access time could be serious obstacles. This sounds more like a research project. Konrad.
participants (5)
-
Konrad Hinsen -
Marcus Denker -
Serge Stinckwich -
serge.stinckwich＠gmail.com
-
Stephan Eggermont