January 2021 - Pharo-users - lists.pharo.org

Re: NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Richard, Am 07.01.21 um 07:15 schrieb Richard O'Keefe: > You aren't sure what point I was making? exactly, the thread you answered was about a possible bug in NeoCSV parser. Your post was about your doubts about the claim of efficiency on the parser's web site. So you threw in some completely unrelated topic and started by sounding more or less destructive (maybe this is a word too harsh, but I am not a native english speaker... maybe "challenging" is a better word?). I cannot comment on the efficiency of NeoCSV, other than it is fast enough for my use case and it gives me the option of combining reading CSV and producing objects in one run, even if some checks, backpointering, whatever has to be done after the parsing. It has a nice API and is supported quite well. The thread and Sven's reaction underline this last statement quite impressively: my bug was fixed within hours. > How about the one I actually wrote down: > Â What test data was NeoCSV benchmarked with > Â and can I get my hands on it? That is a valid question. It is off-topic in the thread, however. And maybe your tone was a bit less kind than it should be. Nevertheless, the discussion itself is worth its own thread. If the raw speed of reading lots of CSV data is of concern in a use case, we should look for and at alternatives. You are of course free to ask about alternatives, present your measurements or alternative implementation and ask for comments, ideas, all kinds of input. That's what yields progress. > THAT is the point. The data points I showed (and > many others I have not) are not satisfactory to me. Fine and absolutely worth discussing. Maybe in its own discussion thread and started with a friendly invitation for discussion. Your post was more like "Oh, and, by the way, NeoCSV sucks". Maybe unintended, but that is what I read. > I have been searching for CSV test collections. > One site offered 6 files of which only one downloaded. > I found a "benchmark suite" for CSV containing no > actual CSV files. > So where *else* should I look for benchmark data than > associated with a parser people in this community are > generally happy with that is described as "efficient"? So you would like the developers of NeoCSV to provide test data that allows for benchmarking and comparison? A valid point. > > Is it so unreasonable to suspect that my results might > be a fluke?Â Is it bad manners to assume that something > described as efficient has tests showing that? Well, no. It is absolutely okay to ask if a claim like "efficient" can be proven. You are free to present better choices and discuss your definition of efficiency. For me personally, your post sounded a bit like some earlier ones of yours which seemed to have no other point than "I have something better, but I won't show you". Hence my reaction. I may have read something into your post that you haven't written into it. Sorry for that. Joachim > > > > On Wed, 6 Jan 2021 at 22:23, jtuchel(a)objektfabrik.de > <mailto:jtuchel@objektfabrik.de> <jtuchel(a)objektfabrik.de > <mailto:jtuchel@objektfabrik.de>> wrote: > > Richard, > > I am not sure what point you are trying to make here. > You have something cooler and faster? Great, how about sharing? > You could make a faster one when it doesn't convert numbers and > stuff? Great. I guess the time will be spent after parsing in 95% > of the use cases. It depends. And that is exactly what you are > saying. The word efficient means nothing without context. How is > that related to this thread? > > I think this thread mostly shows the strength of a community, > especially when there are members who are active, friendly and > highly motivated. My problem git solved in blazing speed without > me paying anything for it. Just because Sven thought my problem > could be other people's problem as well. > > I am happy with NeoCSV's speed, even if there may be more > lightweigt and faster solutions. Tbh, my main concern with NeoCSV > is not speed, but how well I can understand problems and fix them. > I care about data types on parsing. A non-configurable csv parser > gives me a bunch of dictionaries and Strings. That could be a > waste of cycles and memory once you need the data as objects. > My use case is not importing trillions of records all day, and for > a few hundred or maybe sometimes thousands, it is good/fast enough. > > > Joachim > > > > > > Am 06.01.21 um 05:10 schrieb Richard O'Keefe: >> NeoCSVReader is described as efficient.Â What is that >> in comparison to?Â What benchmark data are used? >> Here are benchmark results measured today. >> (5,000 data line file, 9,145,009 characters). >> Â method Â Â Â Â Â Â Â Â time(ms) >> Â Just read characters Â 410 >> Â CSVDecoder>>next Â Â Â 3415 Â astc's CSV reader (defaults). 1.26 >> x CSVParser >> Â NeoCSVReader>>next Â Â 4798 Â NeoCSVReader (default state). 1.78 >> x CSVParser >> Â CSVParser>>next Â Â Â 2701 Â pared-to-the-bone CSV reader. 1.00 >> reference. >> >> (10,000 data line file, 1,544,836 characters). >> Â method Â Â Â Â Â Â Â Â time(ms) >> Â Just read characters Â Â 93 >> Â CSVDecoder>>next Â Â Â 530 Â astc's CSV reader (defaults). 1.26 >> x CSVParser >> Â NeoCSVReader>>next Â Â 737 Â NeoCSVReader (default state). 1.75 >> x CSVParser >> Â CSVParser>>next Â Â Â Â 421 Â pared-to-the-bone CSV reader. 1.00 >> reference. >> >> CSVParser is just 78 lines and is not customisable.Â It really is >> stripped to pretty much an absolute minimum.Â All of the parsers >> were configured (if that made sense) to return an Array of Strings. >> Many of the CSV files I've worked with use short records instead >> of ending a line with a lot of commas.Â Some of them also have >> the occasional stray comment off to the right, not mentioned in >> the header. >> I've also found it necessary to skip multiple lines at the beginning >> and/or end. (Really, some government agencies seem to have NO idea >> that anyone might want to do more with a CSV file than eyeball it in >> Excel.) >> >> If there is a benchmark suite I can use to improve CSVDecoder, >> I would like to try it out. >> >> On Tue, 5 Jan 2021 at 02:36, jtuchel(a)objektfabrik.de >> <mailto:jtuchel@objektfabrik.de> <jtuchel(a)objektfabrik.de >> <mailto:jtuchel@objektfabrik.de>> wrote: >> >> Happy new year to all of you! May 2021 be an increasingly >> less crazy >> year than 2020... >> >> >> I have a question that sounds a bit strange, but we have two >> effects >> with NeoCSVReader related to wrong definitions of the reader. >> >> One effect is that reading a Stream #upToEnd leads to an >> endless loop, >> the other is that the Reader produces twice as many objects >> as there are >> lines in the file that is being read. >> >> In both scenarios, the reason is that the CSV Reader has a >> wrong number >> of column definitions. >> >> Of course that is my fault: why do I feed a "malformed" CSV >> file to poor >> NeoCSVReader? >> >> Let me explain: we have a few import interfaces which end >> users can >> define using a more or less nice assistant in our >> Application. The CSV >> files they upload to our App come from third parties like >> payment >> providers, banks and other sources. These change their file >> structures >> whenever they feel like it and never tell anybody. So a CSV >> import that >> may have been working for years may one day tear a whole web >> server >> image down because of a wrong number of fieldAccessors. This >> is bad on >> many levels. >> >> You can easily try the doubling effect at home: define a >> working CSV >> Reader and comment out one of the addField: commands before >> you use the >> NeoCSVReader to parse a CSV file. Say your CSV file has 3 >> lines with 4 >> columns each. If you remove one of the fieldAccessors, an >> #upToEnd will >> yoield an Array of 6 objects rather than 3. >> >> I haven't found the reason for the cases where this leads to >> an endless >> loop, but at least this one is clear... >> >> I *guess* this is due to the way #readEndOfLine is >> implemented. It seems >> to not peek forward to the end of the line. I have the gut >> feeling >> #peekChar should peek instead of reading the #next character >> form the >> input Stream, but #peekChar has too many senders to just go >> ahead and >> mess with it ;-) >> >> So I wonder if there are any tried approaches to this problem. >> >> One thing I might do is not use #upToEnd, but read each line >> using >> PositionableStream>>#nextLine and first check each line if >> the number of >> separators matches the number of fieldAccessors minus 1 (and >> go through >> the hoops of handling separators in quoted fields and >> such...). Only if >> that test succeeds, I would then hand a Stream with the whole >> line to >> the reader and do a #next. >> >> This will, however, mean a lot of extra cycles for large >> files. Of >> course I could do this only for some lines, maybe just the >> first one. >> Whatever. >> >> >> But somehow I have the feeling I should get an exception >> telling me the >> line is not compatible to the Reader's definition or such. Or >> #readAtEndOrEndOfLine should just walk the line to the end >> and ignore >> the rest of the line, returnong an incomplete object.... >> >> >> Maybe I am just missing the right setting or switch? What >> best practices >> did you guys come up with for such problems? >> >> >> Thanks in advance, >> >> >> Joachim >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > > -- > ----------------------------------------------------------------------- > Objektfabrik Joachim Tuchelmailto:jtuchel@objektfabrik.de <mailto:jtuchel@objektfabrik.de> > Fliederweg 1http://www.objektfabrik.de <http://www.objektfabrik.de> > D-71640 Ludwigsburghttp://joachimtuchel.wordpress.com <http://joachimtuchel.wordpress.com> > Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 > > -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1

Jan. 7, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by Richard O'Keefe

You aren't sure what point I was making? How about the one I actually wrote down: What test data was NeoCSV benchmarked with and can I get my hands on it? THAT is the point. The data points I showed (and many others I have not) are not satisfactory to me. I have been searching for CSV test collections. One site offered 6 files of which only one downloaded. I found a "benchmark suite" for CSV containing no actual CSV files. So where *else* should I look for benchmark data than associated with a parser people in this community are generally happy with that is described as "efficient"? Is it so unreasonable to suspect that my results might be a fluke? Is it bad manners to assume that something described as efficient has tests showing that? On Wed, 6 Jan 2021 at 22:23, jtuchel(a)objektfabrik.de < jtuchel(a)objektfabrik.de> wrote: > Richard, > > I am not sure what point you are trying to make here. > You have something cooler and faster? Great, how about sharing? > You could make a faster one when it doesn't convert numbers and stuff? > Great. I guess the time will be spent after parsing in 95% of the use > cases. It depends. And that is exactly what you are saying. The word > efficient means nothing without context. How is that related to this thread? > > I think this thread mostly shows the strength of a community, especially > when there are members who are active, friendly and highly motivated. My > problem git solved in blazing speed without me paying anything for it. Just > because Sven thought my problem could be other people's problem as well. > > I am happy with NeoCSV's speed, even if there may be more lightweigt and > faster solutions. Tbh, my main concern with NeoCSV is not speed, but how > well I can understand problems and fix them. I care about data types on > parsing. A non-configurable csv parser gives me a bunch of dictionaries and > Strings. That could be a waste of cycles and memory once you need the data > as objects. > My use case is not importing trillions of records all day, and for a few > hundred or maybe sometimes thousands, it is good/fast enough. > > > Joachim > > > > > > Am 06.01.21 um 05:10 schrieb Richard O'Keefe: > > NeoCSVReader is described as efficient. What is that > in comparison to? What benchmark data are used? > Here are benchmark results measured today. > (5,000 data line file, 9,145,009 characters). > method time(ms) > Just read characters 410 > CSVDecoder>>next 3415 astc's CSV reader (defaults). 1.26 x > CSVParser > NeoCSVReader>>next 4798 NeoCSVReader (default state). 1.78 x > CSVParser > CSVParser>>next 2701 pared-to-the-bone CSV reader. 1.00 reference. > > (10,000 data line file, 1,544,836 characters). > method time(ms) > Just read characters 93 > CSVDecoder>>next 530 astc's CSV reader (defaults). 1.26 x > CSVParser > NeoCSVReader>>next 737 NeoCSVReader (default state). 1.75 x > CSVParser > CSVParser>>next 421 pared-to-the-bone CSV reader. 1.00 reference. > > CSVParser is just 78 lines and is not customisable. It really is > stripped to pretty much an absolute minimum. All of the parsers > were configured (if that made sense) to return an Array of Strings. > Many of the CSV files I've worked with use short records instead > of ending a line with a lot of commas. Some of them also have the > occasional stray comment off to the right, not mentioned in the header. > I've also found it necessary to skip multiple lines at the beginning > and/or end. (Really, some government agencies seem to have NO idea > that anyone might want to do more with a CSV file than eyeball it in > Excel.) > > If there is a benchmark suite I can use to improve CSVDecoder, > I would like to try it out. > > On Tue, 5 Jan 2021 at 02:36, jtuchel(a)objektfabrik.de < > jtuchel(a)objektfabrik.de> wrote: > >> Happy new year to all of you! May 2021 be an increasingly less crazy >> year than 2020... >> >> >> I have a question that sounds a bit strange, but we have two effects >> with NeoCSVReader related to wrong definitions of the reader. >> >> One effect is that reading a Stream #upToEnd leads to an endless loop, >> the other is that the Reader produces twice as many objects as there are >> lines in the file that is being read. >> >> In both scenarios, the reason is that the CSV Reader has a wrong number >> of column definitions. >> >> Of course that is my fault: why do I feed a "malformed" CSV file to poor >> NeoCSVReader? >> >> Let me explain: we have a few import interfaces which end users can >> define using a more or less nice assistant in our Application. The CSV >> files they upload to our App come from third parties like payment >> providers, banks and other sources. These change their file structures >> whenever they feel like it and never tell anybody. So a CSV import that >> may have been working for years may one day tear a whole web server >> image down because of a wrong number of fieldAccessors. This is bad on >> many levels. >> >> You can easily try the doubling effect at home: define a working CSV >> Reader and comment out one of the addField: commands before you use the >> NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 >> columns each. If you remove one of the fieldAccessors, an #upToEnd will >> yoield an Array of 6 objects rather than 3. >> >> I haven't found the reason for the cases where this leads to an endless >> loop, but at least this one is clear... >> >> I *guess* this is due to the way #readEndOfLine is implemented. It seems >> to not peek forward to the end of the line. I have the gut feeling >> #peekChar should peek instead of reading the #next character form the >> input Stream, but #peekChar has too many senders to just go ahead and >> mess with it ;-) >> >> So I wonder if there are any tried approaches to this problem. >> >> One thing I might do is not use #upToEnd, but read each line using >> PositionableStream>>#nextLine and first check each line if the number of >> separators matches the number of fieldAccessors minus 1 (and go through >> the hoops of handling separators in quoted fields and such...). Only if >> that test succeeds, I would then hand a Stream with the whole line to >> the reader and do a #next. >> >> This will, however, mean a lot of extra cycles for large files. Of >> course I could do this only for some lines, maybe just the first one. >> Whatever. >> >> >> But somehow I have the feeling I should get an exception telling me the >> line is not compatible to the Reader's definition or such. Or >> #readAtEndOrEndOfLine should just walk the line to the end and ignore >> the rest of the line, returnong an incomplete object.... >> >> >> Maybe I am just missing the right setting or switch? What best practices >> did you guys come up with for such problems? >> >> >> Thanks in advance, >> >> >> Joachim >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > -- > ----------------------------------------------------------------------- > Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de <jtuchel(a)objektfabrik.de> > Fliederweg 1 http://www.objektfabrik.de > D-71640 Ludwigsburg http://joachimtuchel.wordpress.com > Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 > > > >

Jan. 7, 2021

Re: is there a better way

by Esteban Maringolo

To avoid doing something like that, and for a web scrapping tool I wrote, I implemented a basic subclass of ZnClient (called ZnCachingClient) that used a disk cache where each key/file was the hash of the requested URL. I haven't used in a while but it should continue to work, in any case, I published it in a Github repository: https://github.com/eMaringolo/zinc-caching-client Best regards, Esteban A. Maringolo On Wed, Jan 6, 2021 at 3:37 PM Richard Sargent <richard.sargent(a)gemtalksystems.com> wrote: > > > > On Wed, Jan 6, 2021 at 10:34 AM Roelof Wobben via Pharo-users <pharo-users(a)lists.pharo.org> wrote: >> >> >> Thanks, >> >> Right now im downloading/fetching the images every time again. >> >> As I see it, the biggest bottleneck is that I have 10 images. >> And for all 10 I fetching the image and the data I could display when a >> user wants it. >> So that will be 20 calls to the api. >> >> So maybe some cache could be handy. >> Any hints how to make a cache in smalltalk. > > > Dictionary coupled with #at:ifAbsentPut: > >> >> Roelof >> >> >> >> Op 6-1-2021 om 19:19 schreef Sven Van Caekenberghe: >> > Roelof, >> > >> > Working with multiple high resolution images, as I believe you are >> > doing, is always going to be a real challenge, performance wise. It >> > just takes time to transfer lots of data. >> > >> > First you have to make sure that you are not doing too much work >> > (double downloads, using too high resolutions for previews or >> > browsing). Also, make sure your ultimate client (the browser) can >> > cache as well if applicable (set modification dates on the response). >> > >> > Next you could cache images locally (on your app server) so that next >> > time you need the same image, you do not need to download it again. Of >> > course, this only helps if your hit rate is higher than zero (if you >> > actually ask for the same image multiple times). >> > >> > It is also possible to do multiple download requests concurrently: if >> > the other end is fast enough, that can certainly help. >> > >> > HTH, >> > >> > Sven >> > >> >> On 6 Jan 2021, at 18:11, Roelof Wobben via Pharo-users >> >> <pharo-users(a)lists.pharo.org> wrote: >> >> >> >> >> >> I did it on the root document and see this : >> >> >> >> <nkdkdknfekaflcfc.png> >> >> >> >> So as far as I see it , The most time it taken by getting all the >> >> data from all the 10 images. >> >> >> >> I hope someone can look at me if im on the right track and will help >> >> me to figure out faster ways to achieve the same >> >> >> >> Roelof >> >> >> >> >> >> >> >> Op 5-1-2021 om 05:16 schreef Richard O'Keefe: >> >>> Before you take another step, explore the root document. >> >>> >> >>> Profiling is easy. >> >>> Open a Playground. >> >>> Type an expression such as >> >>> 3 tinyBenchmarks >> >>> Right click and select 'Profile it'. >> >>> >> >>> More generally, in a browser, look at the "Tool - Profilers" >> >>> class category. The classic approach was >> >>> MessageTally spyOn: [3 tinyBenchmarks] >> >>> If I understand correctly, 'Profile it' uses TimeProfiler, >> >>> which has a nicer interface. (This is in Pharo 8.) >> >>> >> >>> >> >>> On Sun, 3 Jan 2021 at 23:03, Roelof Wobben <r.wobben(a)home.nl> wrote: >> >>> I want that the code fetches a url and some data from the >> >>> Rijksmuseaum api. >> >>> And as far as I see it the second it not pointless because it >> >>> getting more detailed info about the painting as in the first get. >> >>> >> >>> I did not profiled it because I never learned how to do that in Pharo. >> >>> >> >>> Roelof >> >>> >> >>> >> >>> >> >>> Op 3-1-2021 om 01:09 schreef Richard O'Keefe: >> >>>> What do you want the code to do? >> >>>> Have you profiled the code to see where the time is going? >> >>>> >> >>>> A quick look at the code shows >> >>>> - Paintings does one web get >> >>>> - each Painting does two more web gets >> >>>> ! and the first of those seems to be pretty pointless, >> >>>> as it refetches an object that Paintings already fetched >> >>>> and just looked at. >> >>>> >> >>>> >> >>>> >> >>>> On Sun, 3 Jan 2021 at 01:16, Roelof Wobben via Pharo-users >> >>>> <pharo-users(a)lists.pharo.org> wrote: >> >>>> Hello, >> >>>> >> >>>> I have now this code : https://github.com/RoelofWobben/Rijksmuseam >> >>>> >> >>>> but it seems to be slow. >> >>>> >> >>>> Can anyone help me with a way I can use a sort of cache so the page >> >>>> looks first at the cache if a image is there . >> >>>> If so, take the image from there , if not , ask the api for the url of >> >>>> the image. >> >>>> >> >>>> Roelof

Jan. 6, 2021

Re: is there a better way

by Roelof Wobben

Jan. 6, 2021

Re: is there a better way

by Richard Sargent

On Wed, Jan 6, 2021 at 10:34 AM Roelof Wobben via Pharo-users < pharo-users(a)lists.pharo.org> wrote: > > Thanks, > > Right now im downloading/fetching the images every time again. > > As I see it, the biggest bottleneck is that I have 10 images. > And for all 10 I fetching the image and the data I could display when a > user wants it. > So that will be 20 calls to the api. > > So maybe some cache could be handy. > Any hints how to make a cache in smalltalk. > Dictionary coupled with #at:ifAbsentPut: > Roelof > > > > Op 6-1-2021 om 19:19 schreef Sven Van Caekenberghe: > > Roelof, > > > > Working with multiple high resolution images, as I believe you are > > doing, is always going to be a real challenge, performance wise. It > > just takes time to transfer lots of data. > > > > First you have to make sure that you are not doing too much work > > (double downloads, using too high resolutions for previews or > > browsing). Also, make sure your ultimate client (the browser) can > > cache as well if applicable (set modification dates on the response). > > > > Next you could cache images locally (on your app server) so that next > > time you need the same image, you do not need to download it again. Of > > course, this only helps if your hit rate is higher than zero (if you > > actually ask for the same image multiple times). > > > > It is also possible to do multiple download requests concurrently: if > > the other end is fast enough, that can certainly help. > > > > HTH, > > > > Sven > > > >> On 6 Jan 2021, at 18:11, Roelof Wobben via Pharo-users > >> <pharo-users(a)lists.pharo.org> wrote: > >> > >> > >> I did it on the root document and see this : > >> > >> <nkdkdknfekaflcfc.png> > >> > >> So as far as I see it , The most time it taken by getting all the > >> data from all the 10 images. > >> > >> I hope someone can look at me if im on the right track and will help > >> me to figure out faster ways to achieve the same > >> > >> Roelof > >> > >> > >> > >> Op 5-1-2021 om 05:16 schreef Richard O'Keefe: > >>> Before you take another step, explore the root document. > >>> > >>> Profiling is easy. > >>> Open a Playground. > >>> Type an expression such as > >>> 3 tinyBenchmarks > >>> Right click and select 'Profile it'. > >>> > >>> More generally, in a browser, look at the "Tool - Profilers" > >>> class category. The classic approach was > >>> MessageTally spyOn: [3 tinyBenchmarks] > >>> If I understand correctly, 'Profile it' uses TimeProfiler, > >>> which has a nicer interface. (This is in Pharo 8.) > >>> > >>> > >>> On Sun, 3 Jan 2021 at 23:03, Roelof Wobben <r.wobben(a)home.nl> wrote: > >>> I want that the code fetches a url and some data from the > >>> Rijksmuseaum api. > >>> And as far as I see it the second it not pointless because it > >>> getting more detailed info about the painting as in the first get. > >>> > >>> I did not profiled it because I never learned how to do that in Pharo. > >>> > >>> Roelof > >>> > >>> > >>> > >>> Op 3-1-2021 om 01:09 schreef Richard O'Keefe: > >>>> What do you want the code to do? > >>>> Have you profiled the code to see where the time is going? > >>>> > >>>> A quick look at the code shows > >>>> - Paintings does one web get > >>>> - each Painting does two more web gets > >>>> ! and the first of those seems to be pretty pointless, > >>>> as it refetches an object that Paintings already fetched > >>>> and just looked at. > >>>> > >>>> > >>>> > >>>> On Sun, 3 Jan 2021 at 01:16, Roelof Wobben via Pharo-users > >>>> <pharo-users(a)lists.pharo.org> wrote: > >>>> Hello, > >>>> > >>>> I have now this code : https://github.com/RoelofWobben/Rijksmuseam > >>>> > >>>> but it seems to be slow. > >>>> > >>>> Can anyone help me with a way I can use a sort of cache so the page > >>>> looks first at the cache if a image is there . > >>>> If so, take the image from there , if not , ask the api for the url of > >>>> the image. > >>>> > >>>> Roelof >

Jan. 6, 2021

Re: is there a better way

by Roelof Wobben

Thanks, Right now im downloading/fetching the images every time again. As I see it, the biggest bottleneck is that I have 10 images. And for all 10 I fetching the image and the data I could display when a user wants it. So that will be 20 calls to the api. So maybe some cache could be handy. Any hints how to make a cache in smalltalk. Roelof Op 6-1-2021 om 19:19 schreef Sven Van Caekenberghe: > Roelof, > > Working with multiple high resolution images, as I believe you are > doing, is always going to be a real challenge, performance wise. It > just takes time to transfer lots of data. > > First you have to make sure that you are not doing too much work > (double downloads, using too high resolutions for previews or > browsing). Also, make sure your ultimate client (the browser) can > cache as well if applicable (set modification dates on the response). > > Next you could cache images locally (on your app server) so that next > time you need the same image, you do not need to download it again. Of > course, this only helps if your hit rate is higher than zero (if you > actually ask for the same image multiple times). > > It is also possible to do multiple download requests concurrently: if > the other end is fast enough, that can certainly help. > > HTH, > > Sven > >> On 6 Jan 2021, at 18:11, Roelof Wobben via Pharo-users >> <pharo-users(a)lists.pharo.org> wrote: >> >> >> I did it on the root document and see this : >> >> <nkdkdknfekaflcfc.png> >> >> So as far as I see it , The most time it taken by getting all the >> data from all the 10 images. >> >> I hope someone can look at me if im on the right track and will help >> me to figure out faster ways to achieve the same >> >> Roelof >> >> >> >> Op 5-1-2021 om 05:16 schreef Richard O'Keefe: >>> Before you take another step, explore the root document. >>> >>> Profiling is easy. >>> Open a Playground. >>> Type an expression such as >>> 3 tinyBenchmarks >>> Right click and select 'Profile it'. >>> >>> More generally, in a browser, look at the "Tool - Profilers" >>> class category. The classic approach was >>> MessageTally spyOn: [3 tinyBenchmarks] >>> If I understand correctly, 'Profile it' uses TimeProfiler, >>> which has a nicer interface. (This is in Pharo 8.) >>> >>> >>> On Sun, 3 Jan 2021 at 23:03, Roelof Wobben <r.wobben(a)home.nl> wrote: >>> I want that the code fetches a url and some data from the >>> Rijksmuseaum api. >>> And as far as I see it the second it not pointless because it >>> getting more detailed info about the painting as in the first get. >>> >>> I did not profiled it because I never learned how to do that in Pharo. >>> >>> Roelof >>> >>> >>> >>> Op 3-1-2021 om 01:09 schreef Richard O'Keefe: >>>> What do you want the code to do? >>>> Have you profiled the code to see where the time is going? >>>> >>>> A quick look at the code shows >>>> - Paintings does one web get >>>> - each Painting does two more web gets >>>> ! and the first of those seems to be pretty pointless, >>>> as it refetches an object that Paintings already fetched >>>> and just looked at. >>>> >>>> >>>> >>>> On Sun, 3 Jan 2021 at 01:16, Roelof Wobben via Pharo-users >>>> <pharo-users(a)lists.pharo.org> wrote: >>>> Hello, >>>> >>>> I have now this code : https://github.com/RoelofWobben/Rijksmuseam >>>> >>>> but it seems to be slow. >>>> >>>> Can anyone help me with a way I can use a sort of cache so the page >>>> looks first at the cache if a image is there . >>>> If so, take the image from there , if not , ask the api for the url of >>>> the image. >>>> >>>> Roelof

Jan. 6, 2021

Re: is there a better way

by Sven Van Caekenberghe

Roelof, Working with multiple high resolution images, as I believe you are doing, is always going to be a real challenge, performance wise. It just takes time to transfer lots of data. First you have to make sure that you are not doing too much work (double downloads, using too high resolutions for previews or browsing). Also, make sure your ultimate client (the browser) can cache as well if applicable (set modification dates on the response). Next you could cache images locally (on your app server) so that next time you need the same image, you do not need to download it again. Of course, this only helps if your hit rate is higher than zero (if you actually ask for the same image multiple times). It is also possible to do multiple download requests concurrently: if the other end is fast enough, that can certainly help. HTH, Sven > On 6 Jan 2021, at 18:11, Roelof Wobben via Pharo-users <pharo-users(a)lists.pharo.org> wrote: > > > I did it on the root document and see this : > > <nkdkdknfekaflcfc.png> > > So as far as I see it , The most time it taken by getting all the data from all the 10 images. > > I hope someone can look at me if im on the right track and will help me to figure out faster ways to achieve the same > > Roelof > > > > Op 5-1-2021 om 05:16 schreef Richard O'Keefe: >> Before you take another step, explore the root document. >> >> Profiling is easy. >> Open a Playground. >> Type an expression such as >> 3 tinyBenchmarks >> Right click and select 'Profile it'. >> >> More generally, in a browser, look at the "Tool - Profilers" >> class category. The classic approach was >> MessageTally spyOn: [3 tinyBenchmarks] >> If I understand correctly, 'Profile it' uses TimeProfiler, >> which has a nicer interface. (This is in Pharo 8.) >> >> >> On Sun, 3 Jan 2021 at 23:03, Roelof Wobben <r.wobben(a)home.nl> wrote: >> I want that the code fetches a url and some data from the Rijksmuseaum api. >> And as far as I see it the second it not pointless because it getting more detailed info about the painting as in the first get. >> >> I did not profiled it because I never learned how to do that in Pharo. >> >> Roelof >> >> >> >> Op 3-1-2021 om 01:09 schreef Richard O'Keefe: >>> What do you want the code to do? >>> Have you profiled the code to see where the time is going? >>> >>> A quick look at the code shows >>> - Paintings does one web get >>> - each Painting does two more web gets >>> ! and the first of those seems to be pretty pointless, >>> as it refetches an object that Paintings already fetched >>> and just looked at. >>> >>> >>> >>> On Sun, 3 Jan 2021 at 01:16, Roelof Wobben via Pharo-users <pharo-users(a)lists.pharo.org> wrote: >>> Hello, >>> >>> I have now this code : https://github.com/RoelofWobben/Rijksmuseam >>> >>> but it seems to be slow. >>> >>> Can anyone help me with a way I can use a sort of cache so the page >>> looks first at the cache if a image is there . >>> If so, take the image from there , if not , ask the api for the url of >>> the image. >>> >>> Roelof >> >

Jan. 6, 2021

Re: is there a better way

by Roelof Wobben

Jan. 6, 2021

Re: [Pharo @ FOSDEM 2021] Stand

by Santiago Bragagnolo

Hello everybody. We have been accepted to have a virtual stand in FOSDEM 2021. The idea is to start gathering information to show in the stand about Pharo. Besides the pharo graphical image related content, we would like to offer also success stories, and also videos related to your tools / fwks / business etc, from the point of view of how pharo was a good choice, how it was helpful. Like this we can gather stories that promote your things and pharo at the same time. Also I would like to ask to all those persons doing research on pharo to have some kind of video equivalent to a poster presentation, remarking the interest of using pharo. And to the people that works on pharo core, if you have videos presenting features or assets such as how to build tests, how pharo is integrated with sisop, how it is rendered, how it is interpreted, jited, etc. We could have a track on youtube or similar presenting many features :). Of course if you want to participate actively on the organization of the stand you are more than welcome. Please send me a private message and we may build some communication group to share ideas and organize. Thanks a lot for your attention :) Santiago El miÃ©, 6 ene 2021 a las 15:53, Santiago Bragagnolo (< santiagobragagnolo(a)gmail.com>) escribiÃ³: > We applied to participate in FOSDEM 2021, and we were accepted :)! > > ======================================= > Hello team behind Pharo, > > Your submission for a stand at FOSDEM 2021 has been accepted. > > We welcome you to FOSDEM 2021, our first fully digital edition. > > As a reminder, this is what you can expect this year. > - A spot on our exhibition website (stands.fosdem.org) > - A Matrix chatroom. > - A place to store video's. > > The exhibition website is a static site generated by hugo (gohugo.io). We > have set the theme and the structure, and you can add as many pages as you > want. The code for the website currently lives at > https://github.com/FOSDEM/stands-website . Documentation will be added > shortly (and will be communicated via the list). > > To get started with adding content to the website, follow these steps: > > 1. Create two git repositories (public), one for your content and one for > your static files. Provide us with both links and make sure we can pull > anonymously. > 2. Add pages in any format Hugo accepts. Note that you cannot add more > levels (if you want a deeper structure, ask us). > 3. Once every hour a script will rebuild the website. > 4. You can add links to your own website(s) as well, but we ask that you > keep stand-related things on our stands website. > > We'll provide more information on where to upload the video's later this > month. It is up to you if you wish to create them, but we think it will add > value to your stand. > > More details on the chatroom will also be made available in the coming > week. > > This week, we'll publish the list of accepted stands on our website, and > add the primary and secondary contact to standholders(a)lists.fosdem.org > for further communication. By the beginning of next week, the exhibition > website will be put online (behind authentication as it is not public yet). > As soon as the infrastructure is in place, the video site and the Matrix > chatroom will be configured. > > We'll send further updates to the standholders mailing list. Please bear > with us, this is new for us as well ;-) > > We hope to see you at FOSDEM! > > Kind regards, > > The FOSDEM Stands Team >

Jan. 6, 2021

[Pharo @ FOSDEM 2021] Stand

by Santiago Bragagnolo

We applied to participate in FOSDEM 2021, and we were accepted :)! ======================================= Hello team behind Pharo, Your submission for a stand at FOSDEM 2021 has been accepted. We welcome you to FOSDEM 2021, our first fully digital edition. As a reminder, this is what you can expect this year. - A spot on our exhibition website (stands.fosdem.org) - A Matrix chatroom. - A place to store video's. The exhibition website is a static site generated by hugo (gohugo.io). We have set the theme and the structure, and you can add as many pages as you want. The code for the website currently lives at https://github.com/FOSDEM/stands-website . Documentation will be added shortly (and will be communicated via the list). To get started with adding content to the website, follow these steps: 1. Create two git repositories (public), one for your content and one for your static files. Provide us with both links and make sure we can pull anonymously. 2. Add pages in any format Hugo accepts. Note that you cannot add more levels (if you want a deeper structure, ask us). 3. Once every hour a script will rebuild the website. 4. You can add links to your own website(s) as well, but we ask that you keep stand-related things on our stands website. We'll provide more information on where to upload the video's later this month. It is up to you if you wish to create them, but we think it will add value to your stand. More details on the chatroom will also be made available in the coming week. This week, we'll publish the list of accepted stands on our website, and add the primary and secondary contact to standholders(a)lists.fosdem.org for further communication. By the beginning of next week, the exhibition website will be put online (behind authentication as it is not public yet). As soon as the infrastructure is in place, the video site and the Matrix chatroom will be configured. We'll send further updates to the standholders mailing list. Please bear with us, this is new for us as well ;-) We hope to see you at FOSDEM! Kind regards, The FOSDEM Stands Team

Jan. 6, 2021