Pharo-users

By thread

pharo-users@lists.pharo.org

January 2021

38 participants
140 messages

Re: NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Hi Sven, all I can say is: wow. I have no words. I will have to learn a bit about Pharo and github real quick now in order to try your changes.... Thank you very much. I'll give you feedback as fast as I can. (And forget my questions about #readAtEndOrEndOfLine. I somhow didn't understand it is expected to return a Boolean. Not sure why. I thought of 'read' as a command, not a question in simple past..., so I thought its job should be to read the rest of the line if we're not there yet) Joachim Am 05.01.21 um 17:49 schrieb Sven Van Caekenberghe: > Hi Joachim, > > Have a look at the following commit: > > https://github.com/svenvc/NeoCSV/commit/a3d6258c28138fe3b15aa03ae71cf1e0770… > > and specifically the added unit tests. These should help clarify the new behaviour. > > If anything is not clear, please ask. > > HTH, > > Sven > >> On 5 Jan 2021, at 08:49, jtuchel(a)objektfabrik.de wrote: >> >> Sven, >> >> first of all thanks a lot for taking your time with this! >> >> Your test case is so beautifully small I can't believe it ;-) >> >> While I think some kind of validation could help with parsing CSV, I remember reading your comment on this in some other discussion long ago. You wrote you don't see it as a responsibility of a parser and that you wouldn't want to add this to NeoCSV. I must say I tend to agree mostly. Whatever you do at parsing can only cover part of the problems related to validation. There will be checks that require access to other fields from the same line, or some object that will be the owner of the Collection that you are just importing, so a lot of validation must be done after parsing anyways. >> >> So I think we can mostly ignore the validation part. Whatever a reader will do, it will not be good enough. >> >> A nice way of exposing conversion errors for fields created with #addField:converter: would help a lot, however. >> >> I am glad you agree on the underflow bug. This is more a question of well-formedness than of validation. If a reader finds out it doesn't fit for a file structure, it should tell the user/developer about it or at least gracefully return some more or less incomplete object resembling what it could parse. But it shouldn't cross line borders and return a wrong number of objects. >> >> >> I will definitely continue my hunt for the endless loop. It is not an ideal situation if one user of our Seaside Application completely blocks an image that may be serving a few other users by just using a CVS parser that doesn't fit with the file. I suspect this has to do with #readEndOfLine in some special case of the underflow bug, but cannot prove it yet. But I have a file and parser that reliably goes into an endless loop. I just need to isolate the bare CSV parsing from the whole machinery we've build around NeoCSV reader for these user-defined mappings... I wouldn't be surprised if it is a problem buried somewhere in our preparations in building a parser from user-defined data... I will report my progress here, I promise! >> >> >> One question I keep thinking about in NeoCSV: You implemented a method called #peekChar, but it doesn't #peek. It buffers a character and does read the #next character. I tried replacing the #next with #peek, but that is definitely a shortcut to 100% CPU, because #peekChar is used a lot, not only for consuming an "unmapped remainder" of a line... I somehow have the feeling that at least in #readEndOfLine the next char should bee peeked instead of consumed in order to find out if it's workload or part of the crlf/lf... >> Shouldn't a reader step forward by using #peek to see whether there is more data after all fieldAccessors have been applied to the line (see #readNextRecordAsObject)? Otoh, at one point the reader has to skip to the next line, so I am not sure if peek has any place here... I need to debug a little more to understand... >> >> >> >> Joachim >> >> >> >> >> >> >> Am 04.01.21 um 20:57 schrieb Sven Van Caekenberghe: >>> Hi Joachim, >>> >>> Thanks for the detailed feedback. This is most helpful. I need to think more about this and experiment a bit. This is what I came up with in a Workspace/Playground: >>> >>> input := 'foo,1 >>> bar,2 >>> foobar,3'. >>> >>> (NeoCSVReader on: input readStream) upToEnd. >>> (NeoCSVReader on: input readStream) addField; upToEnd. >>> (NeoCSVReader on: input readStream) addField; addField; addField; upToEnd. >>> >>> (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; upToEnd. >>> (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. >>> (NeoCSVReader on: input readStream) recordClass: Dictionary; emptyFieldValue: #passNil; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. >>> >>> In my opinion there are two distinct issues: >>> >>> 1. what to do when you define a specific number of fields to be read and there are not enough of them in the input (underflow), or there are too many of them in the input (overflow). >>> >>> it is clear that the underflow case is wrong and a bug that has to be fixed. >>> the overflow case seems OK (resulting in nil fields) >>> >>> 2. to validate the input (a functionality not yet present) >>> >>> this would basically mean to signal an error in the under or overflow case. >>> but wrong type conversions should be errors too. >>> >>> I understand that you want to validate foreign input. >>> >>> It is a pity that you cannot produce an infinite loop example, that would also be useful. >>> >>> That's it for now, I will come back to you. >>> >>> Regards, >>> >>> Sven >>> >>>> On 4 Jan 2021, at 14:46, jtuchel(a)objektfabrik.de wrote: >>>> >>>> Please find attached a small test case to demonstrate what I mean. There is just some nonsense Business Object class and a simple test case in this fileout. >>>> >>>> >>>> Am 04.01.21 um 14:36 schrieb jtuchel(a)objektfabrik.de: >>>>> Happy new year to all of you! May 2021 be an increasingly less crazy year than 2020... >>>>> >>>>> >>>>> I have a question that sounds a bit strange, but we have two effects with NeoCSVReader related to wrong definitions of the reader. >>>>> >>>>> One effect is that reading a Stream #upToEnd leads to an endless loop, the other is that the Reader produces twice as many objects as there are lines in the file that is being read. >>>>> >>>>> In both scenarios, the reason is that the CSV Reader has a wrong number of column definitions. >>>>> >>>>> Of course that is my fault: why do I feed a "malformed" CSV file to poor NeoCSVReader? >>>>> >>>>> Let me explain: we have a few import interfaces which end users can define using a more or less nice assistant in our Application. The CSV files they upload to our App come from third parties like payment providers, banks and other sources. These change their file structures whenever they feel like it and never tell anybody. So a CSV import that may have been working for years may one day tear a whole web server image down because of a wrong number of fieldAccessors. This is bad on many levels. >>>>> >>>>> You can easily try the doubling effect at home: define a working CSV Reader and comment out one of the addField: commands before you use the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 columns each. If you remove one of the fieldAccessors, an #upToEnd will yoield an Array of 6 objects rather than 3. >>>>> >>>>> I haven't found the reason for the cases where this leads to an endless loop, but at least this one is clear... >>>>> >>>>> I *guess* this is due to the way #readEndOfLine is implemented. It seems to not peek forward to the end of the line. I have the gut feeling #peekChar should peek instead of reading the #next character form the input Stream, but #peekChar has too many senders to just go ahead and mess with it ;-) >>>>> >>>>> So I wonder if there are any tried approaches to this problem. >>>>> >>>>> One thing I might do is not use #upToEnd, but read each line using PositionableStream>>#nextLine and first check each line if the number of separators matches the number of fieldAccessors minus 1 (and go through the hoops of handling separators in quoted fields and such...). Only if that test succeeds, I would then hand a Stream with the whole line to the reader and do a #next. >>>>> >>>>> This will, however, mean a lot of extra cycles for large files. Of course I could do this only for some lines, maybe just the first one. Whatever. >>>>> >>>>> >>>>> But somehow I have the feeling I should get an exception telling me the line is not compatible to the Reader's definition or such. Or #readAtEndOrEndOfLine should just walk the line to the end and ignore the rest of the line, returnong an incomplete object.... >>>>> >>>>> >>>>> Maybe I am just missing the right setting or switch? What best practices did you guys come up with for such problems? >>>>> >>>>> >>>>> Thanks in advance, >>>>> >>>>> >>>>> Joachim >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> -- >>>> ----------------------------------------------------------------------- >>>> Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de >>>> Fliederweg 1 http://www.objektfabrik.de >>>> D-71640 Ludwigsburg http://joachimtuchel.wordpress.com >>>> Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >>>> >>>> >>>> <NeoCSVEndlessLoopTest.st> >> >> -- >> ----------------------------------------------------------------------- >> Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de >> Fliederweg 1 http://www.objektfabrik.de >> D-71640 Ludwigsburg http://joachimtuchel.wordpress.com >> Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >> -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1

Jan. 5, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by Sven Van Caekenberghe

Hi Joachim, Have a look at the following commit: https://github.com/svenvc/NeoCSV/commit/a3d6258c28138fe3b15aa03ae71cf1e0770… and specifically the added unit tests. These should help clarify the new behaviour. If anything is not clear, please ask. HTH, Sven > On 5 Jan 2021, at 08:49, jtuchel(a)objektfabrik.de wrote: > > Sven, > > first of all thanks a lot for taking your time with this! > > Your test case is so beautifully small I can't believe it ;-) > > While I think some kind of validation could help with parsing CSV, I remember reading your comment on this in some other discussion long ago. You wrote you don't see it as a responsibility of a parser and that you wouldn't want to add this to NeoCSV. I must say I tend to agree mostly. Whatever you do at parsing can only cover part of the problems related to validation. There will be checks that require access to other fields from the same line, or some object that will be the owner of the Collection that you are just importing, so a lot of validation must be done after parsing anyways. > > So I think we can mostly ignore the validation part. Whatever a reader will do, it will not be good enough. > > A nice way of exposing conversion errors for fields created with #addField:converter: would help a lot, however. > > I am glad you agree on the underflow bug. This is more a question of well-formedness than of validation. If a reader finds out it doesn't fit for a file structure, it should tell the user/developer about it or at least gracefully return some more or less incomplete object resembling what it could parse. But it shouldn't cross line borders and return a wrong number of objects. > > > I will definitely continue my hunt for the endless loop. It is not an ideal situation if one user of our Seaside Application completely blocks an image that may be serving a few other users by just using a CVS parser that doesn't fit with the file. I suspect this has to do with #readEndOfLine in some special case of the underflow bug, but cannot prove it yet. But I have a file and parser that reliably goes into an endless loop. I just need to isolate the bare CSV parsing from the whole machinery we've build around NeoCSV reader for these user-defined mappings... I wouldn't be surprised if it is a problem buried somewhere in our preparations in building a parser from user-defined data... I will report my progress here, I promise! > > > One question I keep thinking about in NeoCSV: You implemented a method called #peekChar, but it doesn't #peek. It buffers a character and does read the #next character. I tried replacing the #next with #peek, but that is definitely a shortcut to 100% CPU, because #peekChar is used a lot, not only for consuming an "unmapped remainder" of a line... I somehow have the feeling that at least in #readEndOfLine the next char should bee peeked instead of consumed in order to find out if it's workload or part of the crlf/lf... > Shouldn't a reader step forward by using #peek to see whether there is more data after all fieldAccessors have been applied to the line (see #readNextRecordAsObject)? Otoh, at one point the reader has to skip to the next line, so I am not sure if peek has any place here... I need to debug a little more to understand... > > > > Joachim > > > > > > > Am 04.01.21 um 20:57 schrieb Sven Van Caekenberghe: >> Hi Joachim, >> >> Thanks for the detailed feedback. This is most helpful. I need to think more about this and experiment a bit. This is what I came up with in a Workspace/Playground: >> >> input := 'foo,1 >> bar,2 >> foobar,3'. >> >> (NeoCSVReader on: input readStream) upToEnd. >> (NeoCSVReader on: input readStream) addField; upToEnd. >> (NeoCSVReader on: input readStream) addField; addField; addField; upToEnd. >> >> (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; upToEnd. >> (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. >> (NeoCSVReader on: input readStream) recordClass: Dictionary; emptyFieldValue: #passNil; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. >> >> In my opinion there are two distinct issues: >> >> 1. what to do when you define a specific number of fields to be read and there are not enough of them in the input (underflow), or there are too many of them in the input (overflow). >> >> it is clear that the underflow case is wrong and a bug that has to be fixed. >> the overflow case seems OK (resulting in nil fields) >> >> 2. to validate the input (a functionality not yet present) >> >> this would basically mean to signal an error in the under or overflow case. >> but wrong type conversions should be errors too. >> >> I understand that you want to validate foreign input. >> >> It is a pity that you cannot produce an infinite loop example, that would also be useful. >> >> That's it for now, I will come back to you. >> >> Regards, >> >> Sven >> >>> On 4 Jan 2021, at 14:46, jtuchel(a)objektfabrik.de wrote: >>> >>> Please find attached a small test case to demonstrate what I mean. There is just some nonsense Business Object class and a simple test case in this fileout. >>> >>> >>> Am 04.01.21 um 14:36 schrieb jtuchel(a)objektfabrik.de: >>>> Happy new year to all of you! May 2021 be an increasingly less crazy year than 2020... >>>> >>>> >>>> I have a question that sounds a bit strange, but we have two effects with NeoCSVReader related to wrong definitions of the reader. >>>> >>>> One effect is that reading a Stream #upToEnd leads to an endless loop, the other is that the Reader produces twice as many objects as there are lines in the file that is being read. >>>> >>>> In both scenarios, the reason is that the CSV Reader has a wrong number of column definitions. >>>> >>>> Of course that is my fault: why do I feed a "malformed" CSV file to poor NeoCSVReader? >>>> >>>> Let me explain: we have a few import interfaces which end users can define using a more or less nice assistant in our Application. The CSV files they upload to our App come from third parties like payment providers, banks and other sources. These change their file structures whenever they feel like it and never tell anybody. So a CSV import that may have been working for years may one day tear a whole web server image down because of a wrong number of fieldAccessors. This is bad on many levels. >>>> >>>> You can easily try the doubling effect at home: define a working CSV Reader and comment out one of the addField: commands before you use the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 columns each. If you remove one of the fieldAccessors, an #upToEnd will yoield an Array of 6 objects rather than 3. >>>> >>>> I haven't found the reason for the cases where this leads to an endless loop, but at least this one is clear... >>>> >>>> I *guess* this is due to the way #readEndOfLine is implemented. It seems to not peek forward to the end of the line. I have the gut feeling #peekChar should peek instead of reading the #next character form the input Stream, but #peekChar has too many senders to just go ahead and mess with it ;-) >>>> >>>> So I wonder if there are any tried approaches to this problem. >>>> >>>> One thing I might do is not use #upToEnd, but read each line using PositionableStream>>#nextLine and first check each line if the number of separators matches the number of fieldAccessors minus 1 (and go through the hoops of handling separators in quoted fields and such...). Only if that test succeeds, I would then hand a Stream with the whole line to the reader and do a #next. >>>> >>>> This will, however, mean a lot of extra cycles for large files. Of course I could do this only for some lines, maybe just the first one. Whatever. >>>> >>>> >>>> But somehow I have the feeling I should get an exception telling me the line is not compatible to the Reader's definition or such. Or #readAtEndOrEndOfLine should just walk the line to the end and ignore the rest of the line, returnong an incomplete object.... >>>> >>>> >>>> Maybe I am just missing the right setting or switch? What best practices did you guys come up with for such problems? >>>> >>>> >>>> Thanks in advance, >>>> >>>> >>>> Joachim >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> -- >>> ----------------------------------------------------------------------- >>> Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de >>> Fliederweg 1 http://www.objektfabrik.de >>> D-71640 Ludwigsburg http://joachimtuchel.wordpress.com >>> Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >>> >>> >>> <NeoCSVEndlessLoopTest.st> > > > -- > ----------------------------------------------------------------------- > Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de > Fliederweg 1 http://www.objektfabrik.de > D-71640 Ludwigsburg http://joachimtuchel.wordpress.com > Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >

Jan. 5, 2021

[willow] image cannot be found

by Roelof Wobben

Hello, I try to do the first steps on my journey to make a nice layout in willow but I get stuck. Whatever I try the image which I uploaded is not found. I include a file out in here : WADeploymentAwareFileMetadataLibrary subclass: #MyFileMetadataLibrary Â Â Â instanceVariableNames: '' Â Â Â classVariableNames: '' Â Â Â package: 'PaintingsWeb'! !MyFileMetadataLibrary methodsFor: 'uploaded' stamp: 'RoelofWobben 1/5/2021 13:42'! indexJpgContent Â Â Â ^ #[255 216 255 224 0 16 74 70 73 70 0 1 1 0 0 1 0 1 0 0 255 219 0 132 0 9 6 7 19 19 18 21 18 18 18 22 21 21 23 24 22 21 24 21 21 21 21 21 21 23 22 23 22 22 23 23 23 23 22 24 29 40 32 24 26 37 27 21 21 33 49 33 37 41 43 46 46 46 24 31 51 56 51 45 55 40 45 46 43 1 10 10 10 14 13 14 27 16 16 27 46 37 32 37 45 46 47 45 54 45 45 45 45 47 45 48 45 45 45 50 45 43 47 45 45 45 45 50 45 43 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 255 192 0 17 8 0 169 1 43 3 1 34 0 2 17 1 3 17 1 255 196 0 28 0 0 1 5 1 1 1 0 0 0 0 0 0 0 0 0 0 1 0 2 3 4 5 6 7 8 255 196 0 61 16 0 2 1 2 4 3 6 4 4 5 2 6 3 0 0 0 1 2 17 0 3 4 18 33 49 5 65 81 6 19 34 97 113 129 50 145 161 177 7 66 193 225 20 35 82 209 240 98 241 22 51 114 130 146 162 21 36 147 255 196 0 26 1 0 2 3 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 255 196 0 43 17 0 2 2 1 4 1 3 3 4 3 1 0 0 0 0 0 0 1 2 17 3 18 33 49 65 4 81 97 113 19 34 240 35 129 145 177 161 209 225 50 255 218 0 12 3 1 0 2 17 3 17 0 63 0 241 10 32 80 167 10 137 49 180 169 196 211 77 0 42 84 162 149 49 8 82 38 133 26 0 34 129 164 40 208 50 75 75 173 89 197 91 133 142 99 245 212 125 234 44 10 75 85 190 42 166 20 234 118 31 233 16 72 249 152 170 155 251 146 38 151 218 217 150 105 69 25 161 86 144 21 16 40 1 78 160 0 69 10 57 104 229 164 3 104 129 68 45 18 147 64 80 211 66 158 45 26 89 40 180 20 50 149 60 91 167 182 28 209 104 41 144 210 169 123 131 71 184 245 250 81 104 41 144 138 84 246 180 105 134 128 17 165 74 149 49 10 149 26 17 72 98 165 70 128 160 66 165 74 149 3 8 162 5 54 172 34 105 67 96 136 9 165 64 211 144 80 3 77 41 164 105 83 1 77 42 68 82 20 8 84 166 141 8 160 98 6 166 238 199 90 134 136 52 154 4 104 112 208 51 124 170 223 16 179 32 128 9 34 99 80 35 102 147 215 115 165 82 225 168 75 136 173 183 195 7 70 48 9 6 117 152 35 204 14 90 3 53 150 114 211 146 205 16 139 148 105 28 181 58 40 220 88 36 72 58 242 50 61 141 16 43 83 40 64 2 140 81 138 53 18 64 165 72 81 160 4 162 164 93 12 83 45 239 82 13 233 49 161 192 193 131 81 222 223 202 165 109 88 122 211 241 27 69 70 199 69 116 21 98 211 3 184 168 236 161 234 125 162 165 22 227 153 165 42 4 23 138 143 48 167 50 145 204 212 78 127 213 73 33 177 57 21 87 44 152 169 89 207 245 26 137 78 181 108 85 21 200 105 81 66 167 185 100 128 15 81 34 8 58 73 28 182 219 157 65 21 36 69 161 204 41 180 77 10 0 84 166 149 26 4 10 52 41 80 1 154 149 95 74 134 166 84 210 147 36 136 138 208 20 169 26 144 133 74 105 82 160 66 154 84 98 129 160 98 154 51 66 41 69 2 20 209 2 155 68 26 0 232 248 6 30 109 230 231 175 214 180 175 185 80 202 163 242 199 160 60 254 83 81 240 64 22 204 146 6 131 93 134 211 189 90 195 218 206 95 39 136 149 36 65 27 13 204 251 215 39 36 175 35 111 212 232 40 212 85 122 28 151 20 183 15 185 51 206 32 105 164 8 242 138 173 91 221 167 195 1 149 135 65 185 231 240 144 7 254 53 133 21 208 199 61 80 76 203 146 58 100 208 121 81 20 213 18 41 241 82 34 42 2 166 123 80 1 26 143 177 232 105 145 74 199 67 20 235 82 183 81 76 203 73 90 13 0 76 139 26 159 111 239 229 189 68 247 39 157 57 142 106 148 160 11 202 105 88 193 100 143 47 157 74 35 202 171 160 21 99 47 151 211 247 168 62 70 130 228 14 99 231 81 220 81 59 253 104 173 185 17 26 142 112 125 168 46 251 13 6 209 251 208 128 102 16 41 184 161 200 203 62 41 60 168 113 60 58 91 187 150 219 102 88 6 116 59 242 145 237 81 92 31 228 84 97 164 71 49 183 161 229 250 252 234 212 183 178 13 244 75 126 220 16 63 210 164 250 149 7 245 170 204 42 254 48 197 215 32 236 72 219 250 124 35 236 42 139 211 136 72 109 10 84 141 72 128 169 81 138 20 0 169 82 165 64 133 82 11 149 29 26 6 58 226 193 34 153 20 73 164 40 1 180 102 136 19 66 41 128 166 136 20 130 210 138 0 84 40 249 80 52 0 169 10 81 68 80 35 164 178 85 236 49 26 248 72 244 35 148 114 171 188 24 156 214 160 252 72 235 161 141 64 131 233 176 172 27 1 146 194 220 31 153 217 72 157 192 0 193 31 42 208 225 248 173 109 16 116 23 7 182 109 8 251 86 28 152 221 58 252 216 215 142 116 211 102 142 54 208 188 185 32 102 152 7 156 186 128 191 251 149 249 87 34 5 107 241 46 34 235 113 145 124 37 76 78 231 73 31 173 102 95 62 38 61 117 249 235 250 213 152 34 227 18 25 101 169 216 194 35 90 124 80 229 79 84 32 9 4 3 177 32 193 244 53 107 32 62 211 199 41 7 113 215 247 243 163 118 212 65 6 84 236 127 67 208 211 42 91 87 35 125 65 220 127 155 31 58 137 37 238 66 69 8 214 167 187 110 53 153 83 177 253 15 67 82 226 240 12 182 197 204 195 112 8 27 137 231 62 224 123 209 169 5 50 181 148 220 249 126 162 172 56 251 84 88 126 127 231 49 83 98 12 2 15 149 69 242 53 193 92 169 4 123 85 148 178 206 85 86 89 137 80 160 68 146 76 0 62 117 17 109 189 43 99 10 253 197 190 244 255 0 204 112 69 161 205 87 102 185 239 36 15 115 74 77 130 69 44 102 23 35 21 205 37 78 86 35 81 32 193 131 204 85 59 182 161 183 53 47 123 42 32 78 158 212 219 118 89 181 58 13 164 232 167 211 175 181 10 215 32 247 42 222 80 54 52 184 126 25 153 214 20 144 24 73 228 4 141 206 213 96 50 46 129 115 145 205 180 95 101 26 159 115 237 75 13 125 154 234 23 58 36 184 81 1 70 85 44 33 70 131 106 181 55 68 105 88 110 219 73 44 73 114 196 152 93 23 83 48 91 115 236 61 234 183 18 88 114 160 0 4 104 7 80 42 108 56 232 117 17 30 187 254 223 42 171 121 201 36 157 230 154 228 36 214 146 6 20 40 147 66 166 84 72 145 20 195 66 141 0 3 70 133 42 4 42 52 41 225 40 25 38 46 206 86 32 84 20 251 175 38 105 148 46 1 136 26 64 210 165 76 13 78 21 133 15 111 19 35 85 182 140 190 71 190 182 9 255 0 196 181 43 56 60 224 133 4 145 200 2 79 174 155 84 252 14 228 89 197 176 0 255 0 41 68 121 53 212 79 166 96 125 171 115 133 180 219 5 64 4 145 49 164 70 227 210 179 101 155 137 167 12 35 46 78 45 144 137 158 95 237 173 52 138 215 225 247 114 98 30 219 106 151 11 35 142 69 100 196 30 160 137 6 151 28 224 167 14 192 29 155 85 50 8 34 39 252 222 173 214 174 153 75 131 173 72 200 81 64 86 134 7 132 92 186 27 46 95 12 110 208 117 232 61 233 216 158 15 114 217 139 139 150 64 34 53 38 126 213 45 113 226 196 160 223 8 191 252 122 156 10 217 203 175 122 205 154 6 253 218 0 39 217 190 98 179 147 9 113 80 93 217 79 137 117 248 178 180 29 6 218 131 169 232 107 89 56 106 53 191 9 32 141 74 230 12 15 41 142 181 123 9 129 182 246 157 97 238 176 83 148 38 108 209 18 160 32 243 158 85 159 234 69 108 189 77 79 4 187 125 28 239 21 186 29 141 228 217 163 48 230 173 2 125 137 171 92 79 131 181 188 53 140 65 96 69 217 26 126 83 18 7 202 126 85 143 99 16 84 207 204 114 62 81 90 120 172 121 184 162 216 240 219 85 64 7 47 14 109 122 76 180 72 214 6 181 107 139 141 37 193 153 73 59 177 152 85 6 217 134 1 139 128 103 250 99 151 185 159 111 42 219 227 246 133 132 178 182 196 74 156 235 57 149 254 24 149 146 53 241 25 166 224 176 169 221 155 138 35 34 230 145 205 128 59 243 59 26 117 220 98 58 217 87 70 184 202 8 204 78 92 217 192 133 40 39 104 0 107 172 109 214 157 122 165 183 69 233 37 26 125 156 183 123 226 61 38 180 177 216 6 180 138 237 48 209 200 243 229 235 20 220 118 21 69 208 150 162 98 124 193 230 24 237 35 93 69 111 98 120 118 42 254 23 58 69 196 178 51 50 3 46 128 110 197 78 172 52 27 79 181 78 82 87 18 48 138 169 89 204 93 98 163 41 152 109 78 223 223 122 237 112 152 59 13 132 188 234 253 239 132 202 230 230 163 48 141 178 237 210 185 78 21 195 110 98 174 247 101 192 59 203 68 70 196 233 237 91 92 91 178 183 176 150 209 237 221 102 183 122 3 21 13 111 77 35 48 204 100 120 190 245 28 154 93 43 166 79 18 146 78 90 109 28 213 139 145 39 94 186 111 21 175 141 196 219 24 117 180 1 207 57 156 176 131 35 99 182 196 30 180 120 117 133 200 126 3 4 248 200 241 121 2 4 199 95 122 183 132 70 187 113 112 246 209 77 199 133 89 128 7 169 217 69 41 73 57 113 192 163 22 163 242 80 224 184 21 186 229 156 197 139 64 53 198 234 63 42 47 250 152 233 243 60 169 99 238 53 219 134 227 140 170 96 42 242 84 26 40 85 232 0 175 65 194 118 76 94 13 133 103 40 182 76 59 219 202 13 203 228 120 156 232 70 81 240 168 222 7 157 114 61 163 236 125 252 46 38 221 149 185 222 11 178 81 182 58 17 33 129 208 70 97 175 251 85 112 205 9 201 164 248 36 241 74 49 78 185 49 173 184 0 128 36 204 201 29 64 63 14 220 252 234 27 151 9 98 196 147 203 83 53 161 197 120 85 203 23 150 205 236 160 24 108 202 73 12 167 66 9 16 96 101 168 56 190 25 109 191 128 66 50 252 38 78 86 242 39 159 167 235 86 170 111 228 132 163 37 207 69 11 187 251 107 78 193 221 65 156 182 176 140 162 55 241 64 159 96 90 155 133 196 101 51 164 245 97 48 62 212 220 109 229 147 144 141 70 164 46 93 247 26 69 90 151 69 79 212 122 92 223 161 214 163 196 36 229 35 153 131 228 220 254 122 31 159 74 171 109 226 181 237 92 38 200 98 240 103 46 195 98 68 192 219 109 105 181 165 137 61 72 200 3 149 88 76 19 21 44 4 129 189 27 186 180 229 203 204 149 248 96 147 172 114 244 21 165 129 23 2 176 0 108 73 204 35 194 118 137 162 114 113 68 177 69 75 147 17 133 42 179 140 180 65 219 67 7 65 166 188 170 177 169 167 101 114 84 232 20 169 82 160 136 170 69 106 142 42 85 30 116 49 162 50 41 181 37 205 233 164 208 0 167 32 160 41 212 1 167 131 186 134 219 0 10 192 80 198 73 204 75 239 29 52 26 121 85 222 27 131 104 37 94 36 29 117 208 206 241 34 61 124 249 214 17 184 64 0 192 208 109 211 113 247 173 174 21 143 8 154 180 18 32 16 192 48 130 55 208 192 212 242 235 84 228 77 45 139 177 201 118 81 114 214 249 234 36 204 17 28 244 157 119 214 107 71 139 35 222 183 111 17 4 134 25 88 254 84 117 57 117 228 160 136 61 53 53 155 196 49 115 169 124 238 64 12 198 78 158 46 126 227 229 93 95 4 237 18 45 151 182 19 49 185 111 45 203 110 50 167 120 52 91 138 70 228 137 157 183 62 202 109 197 41 81 102 24 172 141 194 206 82 245 130 8 44 186 65 7 161 137 131 35 113 52 176 249 139 171 16 74 105 48 12 14 64 158 155 86 133 131 30 18 9 2 100 109 7 153 30 219 143 237 91 253 157 198 97 173 155 235 125 25 150 226 12 161 119 144 79 222 119 229 81 150 74 92 23 71 197 119 206 228 252 66 237 183 194 171 133 55 92 64 202 160 72 98 177 48 61 204 87 61 195 59 64 108 222 183 136 75 98 109 206 129 136 204 60 90 54 253 99 65 202 189 3 240 231 138 225 240 246 175 92 186 84 63 132 162 228 147 26 147 148 245 50 23 218 176 187 107 140 254 57 197 192 139 107 32 33 64 3 197 39 82 196 110 118 244 172 216 229 24 189 44 209 147 22 105 173 186 175 228 225 248 198 43 188 196 93 186 66 130 110 51 64 26 31 22 209 53 6 34 242 49 148 82 153 142 169 186 175 76 172 117 34 73 223 111 58 131 20 164 59 3 188 153 174 139 179 61 148 108 84 44 149 102 18 36 120 64 228 79 89 174 132 165 24 70 217 201 140 37 57 82 91 145 224 129 100 22 205 204 163 73 31 213 211 215 218 183 241 61 221 178 169 101 70 82 35 63 230 131 19 169 216 111 160 232 102 179 240 252 46 218 94 189 97 158 90 214 250 2 172 100 8 142 113 173 58 224 181 109 151 190 111 229 47 229 95 204 127 164 249 29 126 213 135 35 82 116 159 253 53 198 226 183 50 112 165 115 220 102 212 78 85 35 152 204 73 63 65 93 159 103 120 78 42 248 39 4 202 178 164 18 100 9 141 87 52 29 99 148 87 15 196 177 139 118 236 97 236 228 12 97 81 53 158 144 188 189 5 104 118 87 182 55 240 110 50 55 132 144 28 17 33 146 100 136 229 207 81 212 245 51 124 241 57 110 85 28 138 59 27 56 62 204 226 240 215 84 221 78 238 65 92 199 225 214 70 142 52 39 156 86 215 31 195 98 111 1 134 177 105 238 11 114 11 32 37 37 192 145 49 176 253 102 189 30 223 104 112 216 171 69 173 186 178 145 17 28 187 178 72 131 190 189 43 42 207 106 240 248 12 21 187 249 131 0 157 216 180 8 206 93 14 88 3 126 91 244 21 153 171 200 104 142 70 177 53 71 152 217 225 183 48 87 82 206 47 9 156 176 104 1 76 185 58 40 82 62 33 48 63 238 138 139 137 112 23 195 97 131 50 220 55 110 180 33 57 128 75 106 79 137 226 64 47 62 16 78 203 60 235 87 143 246 207 29 196 175 11 184 100 132 177 37 109 136 47 44 24 51 157 100 248 76 105 180 207 61 44 99 206 50 238 14 206 107 133 90 238 124 198 55 44 87 77 254 16 163 40 242 39 72 138 178 114 209 207 238 70 9 205 124 7 19 151 11 130 254 39 11 140 186 24 178 42 35 137 14 96 23 149 98 64 31 17 172 251 156 118 254 34 245 187 151 66 247 136 177 10 37 74 143 22 123 100 106 27 77 86 117 3 208 85 172 55 97 241 23 133 187 93 240 22 212 150 150 7 194 76 204 129 161 63 222 186 238 200 246 55 248 32 93 175 203 48 130 153 101 35 80 179 172 131 7 112 121 243 172 207 38 37 27 180 217 115 213 173 116 121 214 92 85 219 139 136 188 140 81 70 101 39 101 83 175 45 129 211 214 161 226 214 210 240 12 224 33 0 252 38 0 241 32 211 72 141 77 118 29 160 226 56 132 183 126 208 76 215 24 144 170 96 42 33 4 120 72 248 180 59 30 115 233 89 124 55 9 133 226 22 23 15 152 97 175 219 7 188 239 34 33 98 119 50 208 217 79 185 246 183 28 155 169 116 189 58 12 216 180 254 251 239 217 231 5 116 3 41 26 206 199 49 219 247 249 214 158 11 130 226 49 140 235 134 180 110 20 80 196 2 160 133 216 64 39 197 237 90 28 64 90 10 108 89 195 230 184 134 13 197 11 150 71 230 14 53 34 118 250 213 222 204 113 87 195 186 189 144 159 196 201 46 21 132 50 147 240 178 245 244 245 157 107 91 200 235 82 70 55 4 158 155 42 112 222 197 103 76 247 110 228 240 204 21 130 12 193 86 157 65 6 178 56 126 5 111 94 40 72 8 179 173 189 36 76 2 38 102 189 27 138 118 128 189 182 185 138 194 169 70 32 183 112 74 93 69 212 179 6 38 25 131 101 240 145 26 29 122 121 242 99 208 98 139 89 5 109 49 152 37 67 101 234 72 208 29 38 6 154 199 157 67 20 242 77 73 223 193 44 145 197 23 20 151 201 13 204 0 76 79 114 27 195 204 180 104 57 130 71 165 109 113 60 65 189 108 132 133 91 100 229 59 151 38 100 47 65 202 177 112 183 70 119 119 185 23 115 64 17 190 224 235 182 241 245 173 47 227 109 219 82 11 3 148 108 8 212 200 0 40 232 0 250 209 145 59 93 180 40 73 36 235 100 204 76 75 55 194 203 16 54 229 235 235 231 85 50 152 158 95 230 198 175 113 53 61 227 107 58 41 49 231 251 17 87 120 23 22 75 107 146 226 43 168 241 0 195 54 186 236 54 7 93 234 235 106 54 145 90 74 82 166 232 194 154 85 111 26 65 34 0 147 36 133 243 51 30 186 213 82 42 197 185 83 84 232 20 166 149 25 160 2 230 77 54 149 42 0 84 86 133 57 5 0 72 228 72 223 206 106 82 70 67 190 156 142 155 239 21 3 239 167 79 126 149 161 194 48 214 179 142 251 85 217 142 176 60 244 222 42 50 116 172 148 83 110 131 192 176 77 113 137 69 206 87 92 163 83 29 64 231 87 187 65 127 187 238 194 232 194 75 2 53 215 127 50 52 216 237 93 110 3 135 90 87 23 48 164 21 97 148 133 34 15 237 251 215 55 219 238 15 113 46 127 16 206 93 92 129 39 117 49 162 192 210 52 211 252 53 142 25 163 147 45 63 207 99 100 177 75 22 43 92 254 110 97 183 22 50 72 64 39 169 157 122 138 177 128 196 0 21 137 235 63 61 189 53 246 172 90 210 225 68 25 83 254 10 213 56 69 71 100 87 135 62 73 228 86 254 14 166 192 208 40 219 73 61 79 246 31 173 11 215 124 64 108 14 222 179 253 226 168 35 181 176 4 202 114 110 104 122 19 208 117 229 87 49 150 102 211 176 140 202 81 134 162 98 94 99 168 208 125 60 171 3 134 246 118 97 150 163 238 191 44 206 237 7 12 146 46 168 232 31 211 145 253 62 85 221 93 182 248 12 53 187 202 193 195 217 10 168 160 102 81 169 12 65 223 83 31 173 115 156 63 22 46 8 43 203 93 37 127 122 118 7 22 46 5 46 100 128 82 36 234 55 219 215 160 165 145 201 197 41 116 83 151 28 35 61 112 127 250 57 91 152 182 24 142 240 16 165 140 249 106 107 67 139 88 123 171 224 76 199 66 66 107 161 49 240 196 239 210 107 67 181 28 38 209 10 45 91 110 247 79 132 12 185 124 244 215 110 92 199 157 96 240 172 125 219 23 51 43 21 101 57 124 193 223 102 216 130 1 214 181 69 169 165 56 242 142 92 150 153 56 203 134 117 125 144 184 184 123 24 134 5 109 98 17 36 139 163 45 208 53 30 12 223 24 59 70 144 99 223 207 166 186 78 214 241 135 197 181 167 186 195 189 91 121 88 141 39 196 72 159 56 143 157 115 102 173 195 10 185 62 89 86 89 112 189 14 151 176 215 28 226 86 216 39 43 102 144 53 217 24 200 232 116 143 122 213 237 94 14 244 48 184 145 38 84 193 137 31 210 121 72 229 231 75 240 121 163 136 44 9 57 31 236 4 255 0 237 94 225 219 27 96 224 245 69 37 124 94 32 52 0 130 209 208 228 205 207 251 86 79 34 90 115 90 233 35 87 143 63 211 209 46 25 243 175 100 49 97 111 162 187 4 66 218 177 144 4 149 220 242 208 26 244 62 41 197 172 119 235 103 13 117 175 192 95 17 101 200 165 67 136 16 33 207 140 147 214 6 181 231 92 86 218 27 217 173 2 18 234 231 11 253 38 74 178 251 48 63 58 213 236 45 128 110 234 96 130 72 145 208 211 242 148 116 60 158 193 133 202 18 208 159 103 166 240 92 93 209 165 200 131 182 145 36 235 200 234 124 170 78 41 196 60 72 3 230 4 192 142 165 213 78 157 117 171 2 198 131 246 172 78 46 177 137 194 157 4 187 207 153 202 79 233 53 193 197 151 92 169 163 171 246 173 251 57 207 197 172 70 34 222 36 20 44 182 251 180 105 85 208 18 88 25 104 211 97 206 184 155 55 29 130 160 26 153 42 192 106 207 164 130 121 235 245 175 79 252 67 199 221 92 117 183 91 133 7 116 15 197 8 210 205 154 68 193 229 243 216 215 25 199 56 189 203 183 195 219 102 12 128 195 13 53 4 72 210 32 105 245 233 93 255 0 30 75 233 197 36 115 243 226 149 107 148 190 17 204 241 11 76 186 23 45 58 157 76 110 70 222 213 78 213 210 164 50 152 35 80 106 108 78 53 158 115 65 36 230 38 53 36 239 61 107 99 129 99 176 200 185 111 0 117 39 84 12 39 202 65 229 233 91 27 113 143 22 115 118 148 185 163 164 225 29 160 91 246 10 199 243 130 145 144 254 99 31 16 63 211 235 183 202 121 62 49 193 110 225 153 88 163 5 133 135 143 9 104 25 160 255 0 212 13 116 15 138 180 89 31 15 104 162 193 5 173 162 130 196 149 129 203 220 242 131 85 248 191 31 55 109 11 18 59 132 49 170 195 51 24 102 204 125 71 32 43 46 54 227 39 165 109 217 166 105 74 43 83 220 228 70 250 15 157 3 87 94 232 23 3 174 83 148 143 15 95 164 69 108 92 227 214 73 241 97 192 59 120 114 199 172 152 173 46 114 85 81 179 62 133 110 217 134 216 185 159 14 235 151 127 173 28 37 181 130 91 96 70 131 70 60 224 114 229 76 189 133 184 9 102 66 6 243 7 46 167 124 219 70 191 90 155 8 227 69 58 2 70 167 109 72 12 125 133 73 241 177 21 238 91 225 224 119 162 68 203 91 83 27 248 167 53 14 45 133 22 243 43 8 105 36 121 106 115 33 30 224 251 121 213 123 119 2 145 58 128 68 145 184 131 50 15 185 171 252 114 253 183 213 1 26 200 4 234 167 40 14 12 235 190 191 42 175 117 36 89 181 51 10 156 46 83 5 42 180 174 195 52 168 176 160 40 1 253 221 59 37 0 245 35 109 81 109 146 164 27 43 58 243 212 125 6 181 121 83 42 31 33 246 21 83 10 163 83 34 117 231 87 239 96 221 240 230 242 106 162 67 13 163 85 212 117 217 166 171 158 237 19 138 219 99 75 135 118 197 212 168 54 83 108 162 9 81 202 52 131 166 159 90 139 143 113 91 152 148 99 114 0 0 149 85 248 71 57 243 53 204 185 210 182 248 110 29 154 203 57 42 84 64 97 62 41 99 148 105 206 161 44 48 131 214 150 229 145 203 57 173 13 156 253 104 240 192 4 30 101 128 242 138 161 117 10 146 14 224 197 110 112 204 33 123 122 15 16 150 215 164 133 251 181 93 150 73 68 94 36 53 100 248 54 184 126 89 25 254 6 75 188 164 73 4 1 239 167 206 168 227 16 164 91 102 254 91 104 31 154 14 158 131 252 137 53 61 204 70 66 182 128 6 21 6 187 201 4 31 109 62 180 252 106 131 101 152 238 20 199 249 214 176 197 181 37 232 206 214 92 113 150 55 92 175 245 186 54 63 14 240 56 85 55 109 226 156 49 49 144 231 107 96 130 63 46 163 156 213 187 189 144 107 118 238 95 178 226 1 112 182 221 67 6 85 38 100 157 140 9 26 114 174 11 130 226 95 196 155 230 129 38 73 7 88 251 215 162 246 127 181 151 25 27 13 117 1 120 62 76 124 244 223 208 15 90 94 68 38 164 218 127 39 63 12 212 160 163 252 28 118 55 136 95 178 169 137 15 145 230 45 229 216 13 65 82 12 248 96 115 231 21 205 221 196 61 215 103 58 177 150 49 245 210 173 241 244 42 249 24 153 89 149 204 8 4 234 34 12 106 15 42 205 178 196 48 32 193 6 65 243 173 216 160 148 108 195 154 77 206 139 152 196 128 173 184 101 6 79 80 74 152 247 6 168 26 216 199 227 94 253 178 92 130 109 196 64 128 16 248 72 3 215 47 204 214 56 21 56 93 110 67 34 87 177 217 254 24 98 22 214 37 239 185 1 109 90 119 36 152 216 169 247 244 174 251 27 219 139 88 252 45 227 101 25 26 216 150 71 133 149 96 218 136 39 77 8 212 215 138 219 196 186 134 1 136 12 50 183 152 144 99 230 41 90 188 194 64 98 1 208 128 72 4 121 198 245 70 95 25 77 182 253 191 193 110 60 218 43 111 83 107 135 227 74 225 115 101 4 165 194 39 152 91 138 15 221 79 206 183 251 48 66 221 14 116 4 146 14 192 131 175 202 170 190 13 71 9 12 20 6 102 82 72 26 159 230 0 36 243 222 141 156 57 182 168 73 211 41 95 61 4 125 207 222 179 121 26 103 22 189 91 70 216 65 195 77 250 38 122 190 31 18 172 162 15 79 218 176 56 253 225 222 225 68 193 55 160 72 211 225 96 192 142 71 93 142 181 165 131 197 91 75 86 230 234 137 80 126 53 6 50 141 34 153 135 192 38 55 19 104 218 184 25 44 248 200 212 234 86 20 142 171 169 218 184 216 176 40 78 247 236 219 41 37 27 49 191 17 48 203 113 237 140 222 52 89 203 213 72 98 218 242 229 92 38 41 213 216 247 104 124 25 1 27 147 148 100 98 35 113 48 107 209 255 0 19 111 225 109 233 152 181 213 132 113 109 140 133 212 67 157 151 227 34 55 57 189 235 205 109 241 41 240 91 69 180 128 193 85 248 167 171 49 213 185 110 107 179 227 69 172 123 244 96 242 50 41 53 70 77 222 30 20 147 117 130 9 36 46 239 19 208 109 84 239 178 126 64 99 171 29 79 176 208 82 197 169 14 211 174 166 161 174 146 57 140 214 181 124 218 176 172 166 29 156 159 101 211 111 243 122 134 209 13 110 235 22 85 32 171 5 157 88 177 212 40 231 183 218 169 189 210 66 131 178 200 30 228 147 247 162 200 50 131 58 146 194 58 1 150 15 212 252 170 42 52 61 67 84 235 239 79 116 36 102 229 26 253 169 130 155 82 16 252 228 2 1 48 98 71 35 29 122 214 191 103 184 45 251 237 41 134 185 117 6 236 37 80 122 185 129 237 53 169 194 49 152 107 43 101 191 134 254 34 227 1 13 116 255 0 45 12 144 64 65 163 28 195 159 81 94 147 217 94 44 247 239 54 23 17 12 183 80 155 90 4 64 165 115 40 10 4 15 15 188 136 172 153 252 135 5 178 53 98 194 158 237 156 15 106 184 24 195 174 98 171 222 102 19 252 229 112 163 127 249 99 206 100 158 187 86 42 88 204 211 117 114 41 35 54 132 105 183 132 14 128 207 202 183 251 71 193 26 213 210 89 96 130 85 193 230 70 255 0 49 175 185 172 43 151 21 73 70 48 52 50 121 141 212 252 185 122 212 112 206 227 179 178 121 162 148 140 27 137 148 149 232 72 161 86 248 163 171 92 44 173 51 191 73 219 67 206 170 86 196 237 25 26 38 196 172 26 138 173 98 6 181 92 173 40 189 134 192 13 75 53 25 74 32 26 24 12 184 42 123 24 168 82 141 37 9 152 147 161 234 42 7 83 206 155 78 173 17 186 102 149 188 45 146 202 13 232 82 178 73 220 52 19 17 235 2 175 223 199 97 172 38 92 56 55 46 104 123 198 16 170 71 64 117 109 43 158 21 35 174 147 231 21 7 142 222 237 147 83 165 178 27 122 241 118 44 198 88 153 36 243 53 220 112 28 27 132 39 47 198 185 68 192 143 24 105 250 10 226 193 0 169 34 70 146 58 215 84 123 91 104 0 5 167 211 148 129 245 154 163 202 140 229 20 160 141 62 30 88 226 147 148 136 184 190 29 151 20 171 167 252 185 7 145 50 71 222 180 207 14 118 195 176 17 57 76 14 102 53 136 174 99 25 218 27 151 46 173 194 171 225 216 107 180 131 169 246 171 159 241 101 204 133 81 2 152 62 41 152 158 130 42 169 96 205 81 170 181 201 124 124 216 253 246 249 178 215 101 16 27 169 114 52 23 45 134 255 0 185 128 62 241 53 236 124 87 176 216 107 189 235 176 203 112 33 238 220 104 1 208 169 39 156 17 30 134 188 255 0 240 187 135 45 252 53 240 231 123 138 217 191 52 233 207 173 122 5 158 62 246 236 226 30 239 138 47 53 155 74 160 3 149 84 13 122 235 58 214 124 217 63 85 175 64 199 7 244 211 93 158 13 199 109 101 185 152 137 145 204 251 254 191 74 202 83 38 77 111 118 197 50 220 30 24 6 96 19 36 70 144 79 58 194 77 53 174 158 23 120 211 48 229 85 145 162 222 14 224 14 51 24 86 148 111 250 88 68 251 104 125 170 157 196 42 197 78 224 144 125 70 134 131 181 27 143 38 79 63 191 58 177 42 43 148 172 109 17 74 40 83 34 122 143 15 194 163 96 44 35 152 82 108 146 119 221 212 237 237 93 31 19 236 229 181 2 235 130 235 160 146 202 170 1 228 136 53 251 154 200 200 44 224 237 49 19 108 181 132 18 60 90 128 65 143 65 91 24 254 24 95 12 247 77 238 243 45 183 100 50 85 84 133 44 15 204 87 7 51 147 146 174 27 127 217 232 155 138 143 91 36 102 240 222 200 53 171 110 215 8 239 14 102 80 134 28 44 13 35 98 196 128 122 2 119 53 163 217 158 62 86 211 53 187 93 217 67 106 212 52 54 172 84 49 240 192 159 18 249 72 174 71 240 223 26 215 11 171 222 61 233 48 51 153 102 4 107 169 213 182 62 154 86 173 251 253 213 235 246 65 25 86 229 134 32 8 241 103 178 12 249 197 25 163 37 57 69 251 20 70 81 150 52 250 43 118 159 19 222 140 81 97 45 14 38 64 241 43 2 24 251 160 175 47 206 103 67 93 231 111 129 181 117 192 248 110 169 97 27 230 240 3 237 169 174 49 176 154 2 43 163 225 53 244 147 245 49 249 211 83 154 75 148 169 144 226 140 144 122 137 247 214 126 160 212 21 164 184 98 109 107 184 37 151 205 100 43 15 102 200 125 218 182 177 189 131 196 162 91 101 107 119 29 192 38 210 56 239 18 68 130 65 229 253 235 91 201 24 242 204 107 28 165 194 57 67 74 166 197 97 94 219 148 184 165 88 104 84 232 69 70 71 251 212 202 234 129 52 41 69 42 0 215 224 205 157 90 201 223 87 67 231 16 195 220 0 222 182 199 90 244 14 207 119 167 11 103 31 43 155 13 114 34 124 76 138 195 50 199 80 78 131 161 62 85 230 24 28 99 90 116 184 191 18 25 21 232 60 15 128 54 40 93 121 200 141 226 48 66 67 102 82 92 102 48 22 80 25 245 223 158 63 42 42 173 155 60 105 116 122 95 226 48 195 155 31 196 181 178 194 234 41 91 137 26 24 37 88 201 28 185 244 154 241 62 57 195 243 160 40 164 186 158 66 67 91 96 78 156 244 32 159 251 188 171 211 241 28 127 15 107 14 152 107 216 129 125 109 166 81 110 210 135 16 52 241 92 32 43 125 43 154 226 29 188 194 170 148 183 134 95 255 0 66 78 157 66 0 162 178 226 115 83 114 140 75 166 163 163 76 153 229 194 141 79 139 184 25 217 194 229 12 204 192 116 4 204 84 107 110 186 150 96 162 91 141 76 13 82 56 168 72 168 170 27 36 13 82 91 53 10 45 76 4 82 99 68 88 182 147 167 189 65 79 120 167 217 6 63 74 154 217 16 229 144 10 144 190 153 124 231 244 167 93 35 165 61 108 6 18 52 52 95 108 43 162 54 182 221 41 182 237 201 137 0 249 233 237 83 247 165 116 34 150 23 86 50 36 29 233 91 160 162 187 161 6 13 59 47 134 117 153 143 40 171 143 134 27 169 131 210 106 60 53 153 209 129 141 117 163 90 171 30 151 103 167 254 22 218 107 118 216 50 156 175 148 171 6 128 192 29 196 214 239 109 82 208 182 64 92 173 226 185 35 73 36 106 211 16 88 254 158 85 229 60 51 137 98 48 254 27 55 0 89 157 85 91 95 113 246 171 220 87 180 119 239 136 123 132 13 136 72 73 210 53 141 235 151 147 198 147 203 173 61 153 209 135 145 21 143 77 110 102 241 219 153 173 134 45 36 177 203 32 230 32 18 14 177 3 149 97 6 173 11 150 9 231 62 245 10 88 131 173 116 113 212 99 70 25 220 165 101 67 93 47 100 59 60 152 171 153 29 152 44 9 54 226 65 57 180 51 35 72 159 122 202 238 80 245 21 232 63 132 246 50 53 210 12 171 105 211 151 230 157 15 151 189 87 228 101 211 141 184 242 75 14 61 83 73 240 115 127 252 77 187 60 65 172 91 45 114 218 8 102 96 38 89 117 129 239 31 58 212 237 103 9 176 152 67 122 213 148 18 85 115 172 128 73 141 68 253 132 239 88 56 238 32 71 16 107 247 9 129 112 235 151 96 190 21 240 207 32 0 138 221 237 223 105 176 216 187 22 173 218 4 186 176 37 130 20 95 132 130 33 140 147 181 84 214 71 146 15 122 165 101 169 193 66 75 190 140 94 207 241 60 232 248 123 204 206 24 41 76 238 228 2 155 5 4 192 53 212 216 196 247 86 156 41 42 145 44 171 164 243 50 6 254 245 231 2 199 41 134 228 8 223 223 149 118 184 16 237 133 201 113 64 148 96 24 8 36 17 27 207 152 229 81 242 177 38 211 189 139 188 60 142 156 107 116 114 248 59 135 188 55 114 202 151 36 79 63 20 199 173 117 92 37 65 108 75 13 117 178 76 147 32 155 153 180 249 1 237 92 207 6 105 139 100 126 98 118 4 78 94 117 210 246 118 234 155 87 200 222 109 207 93 28 115 253 105 249 123 39 251 127 98 241 183 74 253 255 0 162 239 226 42 247 130 217 1 87 41 128 68 146 67 41 153 249 87 24 60 34 24 215 91 219 139 231 42 68 106 192 157 32 236 218 147 206 184 251 196 154 143 136 191 70 43 162 62 93 125 86 74 58 141 249 87 175 112 158 207 90 180 150 174 150 102 97 174 89 57 164 168 5 129 153 205 161 214 188 90 201 10 234 88 144 178 51 70 241 58 253 43 214 191 226 175 254 195 225 36 69 187 11 145 136 57 217 251 172 239 174 194 1 31 35 79 201 132 154 74 33 227 206 49 110 207 62 237 214 46 222 43 25 114 253 146 217 91 44 102 153 240 168 7 66 52 219 206 176 209 8 16 106 193 168 201 173 49 116 168 203 45 221 144 189 137 170 239 108 138 189 64 138 154 155 34 227 102 125 109 241 190 52 46 100 22 51 219 182 182 209 13 188 218 102 81 25 183 214 68 107 84 59 129 64 216 20 219 139 123 137 90 224 86 49 206 166 73 205 228 218 212 74 210 196 251 211 174 219 232 42 48 34 158 220 160 220 44 102 136 6 154 77 56 61 48 39 160 69 61 233 149 89 48 138 83 74 138 208 3 66 142 148 240 212 40 157 232 228 65 137 222 128 164 40 117 164 2 40 9 147 79 90 2 149 12 7 26 65 205 3 189 42 67 12 158 180 13 42 84 0 105 22 166 154 34 128 19 84 252 43 139 223 178 196 218 114 160 130 188 160 142 123 213 102 164 148 233 53 184 91 78 208 252 228 146 79 61 105 141 108 116 167 210 165 96 91 192 99 218 214 185 17 250 103 19 30 148 220 103 17 185 113 139 51 121 101 26 40 244 21 84 83 105 105 87 100 190 164 170 172 151 135 220 9 112 51 76 115 136 251 29 43 107 133 241 108 61 145 118 22 225 239 35 79 15 35 53 128 104 26 83 198 167 201 40 101 148 22 198 183 26 226 226 251 41 85 42 160 68 24 223 174 149 155 154 162 20 105 198 10 42 145 9 73 201 219 36 15 174 186 249 117 29 43 68 241 227 252 107 98 251 177 226 45 54 228 198 86 182 109 145 154 58 29 226 178 169 166 164 146 21 146 43 105 72 154 104 165 64 10 149 42 84 0 105 77 10 84 0 166 171 222 53 98 160 185 82 137 22 71 78 9 78 21 50 237 82 108 73 31 255 217]! ! !MyFileMetadataLibrary methodsFor: 'uploaded metadata' stamp: 'RoelofWobben 1/5/2021 13:42'! indexJpg Â Â Â ^ WAFileLibraryResource Â Â Â Â Â Â filepath: 'index.jpg' Â Â Â Â Â Â mimeType: WAMimeType imageJpeg Â Â Â Â Â Â cacheDuration: self class defaultCacheDuration Â Â Â Â Â Â contents: (GRDelayedSend receiver: self selector: #indexJpgContent)! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! MyFileMetadataLibrary class Â Â Â instanceVariableNames: ''! !MyFileMetadataLibrary class methodsFor: 'Accessing' stamp: 'RoelofWobben 1/5/2021 13:45'! version Â ^ '0.0.1'! ! !MyFileMetadataLibrary class methodsFor: 'Accessing' stamp: 'RoelofWobben 1/5/2021 13:51'! forDevelopment Â ^ self default ! ! !MyFileMetadataLibrary class methodsFor: 'Accessing' stamp: 'RoelofWobben 1/5/2021 13:46'! libraryName Â Â Â ^ 'MyFileMetadataLibrary'! ! !MyFileMetadataLibrary class methodsFor: 'Accessing' stamp: 'RoelofWobben 1/5/2021 13:51'! forDeployment Â ^ self default ! ! WillowApplication subclass: #TestImages Â Â Â instanceVariableNames: '' Â Â Â classVariableNames: '' Â Â Â package: 'PaintingsWeb'! !TestImages methodsFor: 'as yet unclassified' stamp: 'RoelofWobben 1/5/2021 13:28'! contentView Â ^ [ :canvas | self renderImageOn: canvas ]! ! !TestImages methodsFor: 'as yet unclassified' stamp: 'RoelofWobben 1/4/2021 15:47'! jQueryLibrary Â ^ JQuery3OnlineLibrary default ! ! !TestImages methodsFor: 'as yet unclassified' stamp: 'RoelofWobben 1/5/2021 13:43'! renderImageOn: aCanvas aCanvas render: ( ImageWebView locatedAt: MyFileMetadataLibrary / #indexJpg applying: [ ] ) ! ! !TestImages methodsFor: 'accessing' stamp: 'RoelofWobben 1/4/2021 19:00'! componentSupplierForApplication Â ^ Bootstrap4ComponentSupplier online! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! TestImages class Â Â Â instanceVariableNames: ''! !TestImages class methodsFor: 'accessing' stamp: 'RoelofWobben 1/4/2021 19:02'! applicationName Â ^Â 'Paintings' ! ! !TestImages class methodsFor: 'accessing' stamp: 'RoelofWobben 1/4/2021 19:03'! applicationTitle Â ^Â 'Paintings' ! ! !TestImages class methodsFor: 'accessing' stamp: 'RoelofWobben 1/4/2021 18:02'! handlerName Â ^ 'painting' ! ! Anyone knows what I did wrong here Roelof

Jan. 5, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Sven, first of all thanks a lot for taking your time with this! Your test case is so beautifully small I can't believe it ;-) While I think some kind of validation could help with parsing CSV, I remember reading your comment on this in some other discussion long ago. You wrote you don't see it as a responsibility of a parser and that you wouldn't want to add this to NeoCSV. I must say I tend to agree mostly. Whatever you do at parsing can only cover part of the problems related to validation. There will be checks that require access to other fields from the same line, or some object that will be the owner of the Collection that you are just importing, so a lot of validation must be done after parsing anyways. So I think we can mostly ignore the validation part. Whatever a reader will do, it will not be good enough. A nice way of exposing conversion errors for fields created with #addField:converter: would help a lot, however. I am glad you agree on the underflow bug. This is more a question of well-formedness than of validation. If a reader finds out it doesn't fit for a file structure, it should tell the user/developer about it or at least gracefully return some more or less incomplete object resembling what it could parse. But it shouldn't cross line borders and return a wrong number of objects. I will definitely continue my hunt for the endless loop. It is not an ideal situation if one user of our Seaside Application completely blocks an image that may be serving a few other users by just using a CVS parser that doesn't fit with the file. I suspect this has to do with #readEndOfLine in some special case of the underflow bug, but cannot prove it yet. But I have a file and parser that reliably goes into an endless loop. I just need to isolate the bare CSV parsing from the whole machinery we've build around NeoCSV reader for these user-defined mappings... I wouldn't be surprised if it is a problem buried somewhere in our preparations in building a parser from user-defined data... I will report my progress here, I promise! One question I keep thinking about in NeoCSV: You implemented a method called #peekChar, but it doesn't #peek. It buffers a character and does read the #next character. I tried replacing the #next with #peek, but that is definitely a shortcut to 100% CPU, because #peekChar is used a lot, not only for consuming an "unmapped remainder" of a line... I somehow have the feeling that at least in #readEndOfLine the next char should bee peeked instead of consumed in order to find out if it's workload or part of the crlf/lf... Shouldn't a reader step forward by using #peek to see whether there is more data after all fieldAccessors have been applied to the line (see #readNextRecordAsObject)? Otoh, at one point the reader has to skip to the next line, so I am not sure if peek has any place here... I need to debug a little more to understand... Joachim Am 04.01.21 um 20:57 schrieb Sven Van Caekenberghe: > Hi Joachim, > > Thanks for the detailed feedback. This is most helpful. I need to think more about this and experiment a bit. This is what I came up with in a Workspace/Playground: > > input := 'foo,1 > bar,2 > foobar,3'. > > (NeoCSVReader on: input readStream) upToEnd. > (NeoCSVReader on: input readStream) addField; upToEnd. > (NeoCSVReader on: input readStream) addField; addField; addField; upToEnd. > > (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; upToEnd. > (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. > (NeoCSVReader on: input readStream) recordClass: Dictionary; emptyFieldValue: #passNil; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. > > In my opinion there are two distinct issues: > > 1. what to do when you define a specific number of fields to be read and there are not enough of them in the input (underflow), or there are too many of them in the input (overflow). > > it is clear that the underflow case is wrong and a bug that has to be fixed. > the overflow case seems OK (resulting in nil fields) > > 2. to validate the input (a functionality not yet present) > > this would basically mean to signal an error in the under or overflow case. > but wrong type conversions should be errors too. > > I understand that you want to validate foreign input. > > It is a pity that you cannot produce an infinite loop example, that would also be useful. > > That's it for now, I will come back to you. > > Regards, > > Sven > >> On 4 Jan 2021, at 14:46, jtuchel(a)objektfabrik.de wrote: >> >> Please find attached a small test case to demonstrate what I mean. There is just some nonsense Business Object class and a simple test case in this fileout. >> >> >> Am 04.01.21 um 14:36 schrieb jtuchel(a)objektfabrik.de: >>> Happy new year to all of you! May 2021 be an increasingly less crazy year than 2020... >>> >>> >>> I have a question that sounds a bit strange, but we have two effects with NeoCSVReader related to wrong definitions of the reader. >>> >>> One effect is that reading a Stream #upToEnd leads to an endless loop, the other is that the Reader produces twice as many objects as there are lines in the file that is being read. >>> >>> In both scenarios, the reason is that the CSV Reader has a wrong number of column definitions. >>> >>> Of course that is my fault: why do I feed a "malformed" CSV file to poor NeoCSVReader? >>> >>> Let me explain: we have a few import interfaces which end users can define using a more or less nice assistant in our Application. The CSV files they upload to our App come from third parties like payment providers, banks and other sources. These change their file structures whenever they feel like it and never tell anybody. So a CSV import that may have been working for years may one day tear a whole web server image down because of a wrong number of fieldAccessors. This is bad on many levels. >>> >>> You can easily try the doubling effect at home: define a working CSV Reader and comment out one of the addField: commands before you use the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 columns each. If you remove one of the fieldAccessors, an #upToEnd will yoield an Array of 6 objects rather than 3. >>> >>> I haven't found the reason for the cases where this leads to an endless loop, but at least this one is clear... >>> >>> I *guess* this is due to the way #readEndOfLine is implemented. It seems to not peek forward to the end of the line. I have the gut feeling #peekChar should peek instead of reading the #next character form the input Stream, but #peekChar has too many senders to just go ahead and mess with it ;-) >>> >>> So I wonder if there are any tried approaches to this problem. >>> >>> One thing I might do is not use #upToEnd, but read each line using PositionableStream>>#nextLine and first check each line if the number of separators matches the number of fieldAccessors minus 1 (and go through the hoops of handling separators in quoted fields and such...). Only if that test succeeds, I would then hand a Stream with the whole line to the reader and do a #next. >>> >>> This will, however, mean a lot of extra cycles for large files. Of course I could do this only for some lines, maybe just the first one. Whatever. >>> >>> >>> But somehow I have the feeling I should get an exception telling me the line is not compatible to the Reader's definition or such. Or #readAtEndOrEndOfLine should just walk the line to the end and ignore the rest of the line, returnong an incomplete object.... >>> >>> >>> Maybe I am just missing the right setting or switch? What best practices did you guys come up with for such problems? >>> >>> >>> Thanks in advance, >>> >>> >>> Joachim >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> -- >> ----------------------------------------------------------------------- >> Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de >> Fliederweg 1 http://www.objektfabrik.de >> D-71640 Ludwigsburg http://joachimtuchel.wordpress.com >> Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >> >> >> <NeoCSVEndlessLoopTest.st> -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1

Jan. 5, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Paul, thank you very much for this idea. Your suggestion is probably "good enough" to at least catch errors when the number of columns doesn't match in the whole file or the first row. For my use case, it wouldn't make any difference if the first row contains header information or not. There might be cases where you should check each and every line and produce an error if the number of coumns doesn't match. So I'll definitely take a look at #readHeader and see if I can use this as a guard against maybe 80% of error cases, which is already a great step forward. For now I am still hunting for the endless loop, which I guess is related, but I cannot say for sure yet. Maybe your fix also significantly reduces the risk of these loops. Joachim As a side note: isn't it interesting how many creative ideas people have around what they call "CSV export"? Like a header of 12 nicely formatted text lines that beautifully displays in Excel, but of course doesn't resemble anything near the ida of CSV.... Som even add some column separators at the beginning of header lines to make the text apper in column B or C, but never append any empty fields in their header lines... Am 04.01.21 um 19:23 schrieb Paul DeBruicker: > After instantiating the reader and before doing the reading you can > #readHeader and check that the reader field count and header field count > match. Would that help? > > If the CSV doesn't use headers then you can process the "header" as the > first record and then process the rest of the file. > > > > jtuchel wrote >> Please find attached a small test case to demonstrate what I mean. There >> is just some nonsense Business Object class and a simple test case in >> this fileout. >> >> >> Am 04.01.21 um 14:36 schrieb >> jtuchel@ >> : >>> Happy new year to all of you! May 2021 be an increasingly less crazy >>> year than 2020... >>> >>> >>> I have a question that sounds a bit strange, but we have two effects >>> with NeoCSVReader related to wrong definitions of the reader. >>> >>> One effect is that reading a Stream #upToEnd leads to an endless loop, >>> the other is that the Reader produces twice as many objects as there >>> are lines in the file that is being read. >>> >>> In both scenarios, the reason is that the CSV Reader has a wrong >>> number of column definitions. >>> >>> Of course that is my fault: why do I feed a "malformed" CSV file to >>> poor NeoCSVReader? >>> >>> Let me explain: we have a few import interfaces which end users can >>> define using a more or less nice assistant in our Application. The CSV >>> files they upload to our App come from third parties like payment >>> providers, banks and other sources. These change their file structures >>> whenever they feel like it and never tell anybody. So a CSV import >>> that may have been working for years may one day tear a whole web >>> server image down because of a wrong number of fieldAccessors. This is >>> bad on many levels. >>> >>> You can easily try the doubling effect at home: define a working CSV >>> Reader and comment out one of the addField: commands before you use >>> the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines >>> with 4 columns each. If you remove one of the fieldAccessors, an >>> #upToEnd will yoield an Array of 6 objects rather than 3. >>> >>> I haven't found the reason for the cases where this leads to an >>> endless loop, but at least this one is clear... >>> >>> I *guess* this is due to the way #readEndOfLine is implemented. It >>> seems to not peek forward to the end of the line. I have the gut >>> feeling #peekChar should peek instead of reading the #next character >>> form the input Stream, but #peekChar has too many senders to just go >>> ahead and mess with it ;-) >>> >>> So I wonder if there are any tried approaches to this problem. >>> >>> One thing I might do is not use #upToEnd, but read each line using >>> PositionableStream>>#nextLine and first check each line if the number >>> of separators matches the number of fieldAccessors minus 1 (and go >>> through the hoops of handling separators in quoted fields and >>> such...). Only if that test succeeds, I would then hand a Stream with >>> the whole line to the reader and do a #next. >>> >>> This will, however, mean a lot of extra cycles for large files. Of >>> course I could do this only for some lines, maybe just the first one. >>> Whatever. >>> >>> >>> But somehow I have the feeling I should get an exception telling me >>> the line is not compatible to the Reader's definition or such. Or >>> #readAtEndOrEndOfLine should just walk the line to the end and ignore >>> the rest of the line, returnong an incomplete object.... >>> >>> >>> Maybe I am just missing the right setting or switch? What best >>> practices did you guys come up with for such problems? >>> >>> >>> Thanks in advance, >>> >>> >>> Joachim >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> -- >> ----------------------------------------------------------------------- >> Objektfabrik Joachim Tuchel mailto: >> jtuchel@ >> Fliederweg 1 http://www.objektfabrik.de >> D-71640 Ludwigsburg http://joachimtuchel.wordpress.com >> Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 >> >> >> >> >> NeoCSVEndlessLoopTest.st (2K) >> <http://forum.world.st/attachment/5125853/0/NeoCSVEndlessLoopTest.st> > > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html > -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1

Jan. 5, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by Sven Van Caekenberghe

Hi Joachim, Thanks for the detailed feedback. This is most helpful. I need to think more about this and experiment a bit. This is what I came up with in a Workspace/Playground: input := 'foo,1 bar,2 foobar,3'. (NeoCSVReader on: input readStream) upToEnd. (NeoCSVReader on: input readStream) addField; upToEnd. (NeoCSVReader on: input readStream) addField; addField; addField; upToEnd. (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; upToEnd. (NeoCSVReader on: input readStream) recordClass: Dictionary; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. (NeoCSVReader on: input readStream) recordClass: Dictionary; emptyFieldValue: #passNil; addField: [ :obj :str | obj at: #one put: str]; addField: [ :obj :str | obj at: #two put: str]; addField: [ :obj :str | obj at: #three put: str]; upToEnd. In my opinion there are two distinct issues: 1. what to do when you define a specific number of fields to be read and there are not enough of them in the input (underflow), or there are too many of them in the input (overflow). it is clear that the underflow case is wrong and a bug that has to be fixed. the overflow case seems OK (resulting in nil fields) 2. to validate the input (a functionality not yet present) this would basically mean to signal an error in the under or overflow case. but wrong type conversions should be errors too. I understand that you want to validate foreign input. It is a pity that you cannot produce an infinite loop example, that would also be useful. That's it for now, I will come back to you. Regards, Sven > On 4 Jan 2021, at 14:46, jtuchel(a)objektfabrik.de wrote: > > Please find attached a small test case to demonstrate what I mean. There is just some nonsense Business Object class and a simple test case in this fileout. > > > Am 04.01.21 um 14:36 schrieb jtuchel(a)objektfabrik.de: >> Happy new year to all of you! May 2021 be an increasingly less crazy year than 2020... >> >> >> I have a question that sounds a bit strange, but we have two effects with NeoCSVReader related to wrong definitions of the reader. >> >> One effect is that reading a Stream #upToEnd leads to an endless loop, the other is that the Reader produces twice as many objects as there are lines in the file that is being read. >> >> In both scenarios, the reason is that the CSV Reader has a wrong number of column definitions. >> >> Of course that is my fault: why do I feed a "malformed" CSV file to poor NeoCSVReader? >> >> Let me explain: we have a few import interfaces which end users can define using a more or less nice assistant in our Application. The CSV files they upload to our App come from third parties like payment providers, banks and other sources. These change their file structures whenever they feel like it and never tell anybody. So a CSV import that may have been working for years may one day tear a whole web server image down because of a wrong number of fieldAccessors. This is bad on many levels. >> >> You can easily try the doubling effect at home: define a working CSV Reader and comment out one of the addField: commands before you use the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 columns each. If you remove one of the fieldAccessors, an #upToEnd will yoield an Array of 6 objects rather than 3. >> >> I haven't found the reason for the cases where this leads to an endless loop, but at least this one is clear... >> >> I *guess* this is due to the way #readEndOfLine is implemented. It seems to not peek forward to the end of the line. I have the gut feeling #peekChar should peek instead of reading the #next character form the input Stream, but #peekChar has too many senders to just go ahead and mess with it ;-) >> >> So I wonder if there are any tried approaches to this problem. >> >> One thing I might do is not use #upToEnd, but read each line using PositionableStream>>#nextLine and first check each line if the number of separators matches the number of fieldAccessors minus 1 (and go through the hoops of handling separators in quoted fields and such...). Only if that test succeeds, I would then hand a Stream with the whole line to the reader and do a #next. >> >> This will, however, mean a lot of extra cycles for large files. Of course I could do this only for some lines, maybe just the first one. Whatever. >> >> >> But somehow I have the feeling I should get an exception telling me the line is not compatible to the Reader's definition or such. Or #readAtEndOrEndOfLine should just walk the line to the end and ignore the rest of the line, returnong an incomplete object.... >> >> >> Maybe I am just missing the right setting or switch? What best practices did you guys come up with for such problems? >> >> >> Thanks in advance, >> >> >> Joachim >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > > -- > ----------------------------------------------------------------------- > Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de > Fliederweg 1 http://www.objektfabrik.de > D-71640 Ludwigsburg http://joachimtuchel.wordpress.com > Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 > > > <NeoCSVEndlessLoopTest.st>

Jan. 4, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by Paul DeBruicker

After instantiating the reader and before doing the reading you can #readHeader and check that the reader field count and header field count match. Would that help? If the CSV doesn't use headers then you can process the "header" as the first record and then process the rest of the file. jtuchel wrote > Please find attached a small test case to demonstrate what I mean. There > is just some nonsense Business Object class and a simple test case in > this fileout. > > > Am 04.01.21 um 14:36 schrieb > jtuchel@ > : >> Happy new year to all of you! May 2021 be an increasingly less crazy >> year than 2020... >> >> >> I have a question that sounds a bit strange, but we have two effects >> with NeoCSVReader related to wrong definitions of the reader. >> >> One effect is that reading a Stream #upToEnd leads to an endless loop, >> the other is that the Reader produces twice as many objects as there >> are lines in the file that is being read. >> >> In both scenarios, the reason is that the CSV Reader has a wrong >> number of column definitions. >> >> Of course that is my fault: why do I feed a "malformed" CSV file to >> poor NeoCSVReader? >> >> Let me explain: we have a few import interfaces which end users can >> define using a more or less nice assistant in our Application. The CSV >> files they upload to our App come from third parties like payment >> providers, banks and other sources. These change their file structures >> whenever they feel like it and never tell anybody. So a CSV import >> that may have been working for years may one day tear a whole web >> server image down because of a wrong number of fieldAccessors. This is >> bad on many levels. >> >> You can easily try the doubling effect at home: define a working CSV >> Reader and comment out one of the addField: commands before you use >> the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines >> with 4 columns each. If you remove one of the fieldAccessors, an >> #upToEnd will yoield an Array of 6 objects rather than 3. >> >> I haven't found the reason for the cases where this leads to an >> endless loop, but at least this one is clear... >> >> I *guess* this is due to the way #readEndOfLine is implemented. It >> seems to not peek forward to the end of the line. I have the gut >> feeling #peekChar should peek instead of reading the #next character >> form the input Stream, but #peekChar has too many senders to just go >> ahead and mess with it ;-) >> >> So I wonder if there are any tried approaches to this problem. >> >> One thing I might do is not use #upToEnd, but read each line using >> PositionableStream>>#nextLine and first check each line if the number >> of separators matches the number of fieldAccessors minus 1 (and go >> through the hoops of handling separators in quoted fields and >> such...). Only if that test succeeds, I would then hand a Stream with >> the whole line to the reader and do a #next. >> >> This will, however, mean a lot of extra cycles for large files. Of >> course I could do this only for some lines, maybe just the first one. >> Whatever. >> >> >> But somehow I have the feeling I should get an exception telling me >> the line is not compatible to the Reader's definition or such. Or >> #readAtEndOrEndOfLine should just walk the line to the end and ignore >> the rest of the line, returnong an incomplete object.... >> >> >> Maybe I am just missing the right setting or switch? What best >> practices did you guys come up with for such problems? >> >> >> Thanks in advance, >> >> >> Joachim >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > > -- > ----------------------------------------------------------------------- > Objektfabrik Joachim Tuchel mailto: > jtuchel@ > Fliederweg 1 http://www.objektfabrik.de > D-71640 Ludwigsburg http://joachimtuchel.wordpress.com > Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1 > > > > > NeoCSVEndlessLoopTest.st (2K) > <http://forum.world.st/attachment/5125853/0/NeoCSVEndlessLoopTest.st> -- Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Jan. 4, 2021

Re: NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Please find attached a small test case to demonstrate what I mean. There is just some nonsense Business Object class and a simple test case in this fileout. Am 04.01.21 um 14:36 schrieb jtuchel(a)objektfabrik.de: > Happy new year to all of you! May 2021 be an increasingly less crazy > year than 2020... > > > I have a question that sounds a bit strange, but we have two effects > with NeoCSVReader related to wrong definitions of the reader. > > One effect is that reading a Stream #upToEnd leads to an endless loop, > the other is that the Reader produces twice as many objects as there > are lines in the file that is being read. > > In both scenarios, the reason is that the CSV Reader has a wrong > number of column definitions. > > Of course that is my fault: why do I feed a "malformed" CSV file to > poor NeoCSVReader? > > Let me explain: we have a few import interfaces which end users can > define using a more or less nice assistant in our Application. The CSV > files they upload to our App come from third parties like payment > providers, banks and other sources. These change their file structures > whenever they feel like it and never tell anybody. So a CSV import > that may have been working for years may one day tear a whole web > server image down because of a wrong number of fieldAccessors. This is > bad on many levels. > > You can easily try the doubling effect at home: define a working CSV > Reader and comment out one of the addField: commands before you use > the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines > with 4 columns each. If you remove one of the fieldAccessors, an > #upToEnd will yoield an Array of 6 objects rather than 3. > > I haven't found the reason for the cases where this leads to an > endless loop, but at least this one is clear... > > I *guess* this is due to the way #readEndOfLine is implemented. It > seems to not peek forward to the end of the line. I have the gut > feeling #peekChar should peek instead of reading the #next character > form the input Stream, but #peekChar has too many senders to just go > ahead and mess with it ;-) > > So I wonder if there are any tried approaches to this problem. > > One thing I might do is not use #upToEnd, but read each line using > PositionableStream>>#nextLine and first check each line if the number > of separators matches the number of fieldAccessors minus 1 (and go > through the hoops of handling separators in quoted fields and > such...). Only if that test succeeds, I would then hand a Stream with > the whole line to the reader and do a #next. > > This will, however, mean a lot of extra cycles for large files. Of > course I could do this only for some lines, maybe just the first one. > Whatever. > > > But somehow I have the feeling I should get an exception telling me > the line is not compatible to the Reader's definition or such. Or > #readAtEndOrEndOfLine should just walk the line to the end and ignore > the rest of the line, returnong an incomplete object.... > > > Maybe I am just missing the right setting or switch? What best > practices did you guys come up with for such problems? > > > Thanks in advance, > > > Joachim > > > > > > > > > > > > > > > > > > > -- ----------------------------------------------------------------------- Objektfabrik Joachim Tuchel mailto:jtuchel@objektfabrik.de Fliederweg 1 http://www.objektfabrik.de D-71640 Ludwigsburg http://joachimtuchel.wordpress.com Telefon: +49 7141 56 10 86 0 Fax: +49 7141 56 10 86 1

Jan. 4, 2021

NeoCSVReader and wrong number of fieldAccessors

by jtuchel＠objektfabrik.de

Happy new year to all of you! May 2021 be an increasingly less crazy year than 2020... I have a question that sounds a bit strange, but we have two effects with NeoCSVReader related to wrong definitions of the reader. One effect is that reading a Stream #upToEnd leads to an endless loop, the other is that the Reader produces twice as many objects as there are lines in the file that is being read. In both scenarios, the reason is that the CSV Reader has a wrong number of column definitions. Of course that is my fault: why do I feed a "malformed" CSV file to poor NeoCSVReader? Let me explain: we have a few import interfaces which end users can define using a more or less nice assistant in our Application. The CSV files they upload to our App come from third parties like payment providers, banks and other sources. These change their file structures whenever they feel like it and never tell anybody. So a CSV import that may have been working for years may one day tear a whole web server image down because of a wrong number of fieldAccessors. This is bad on many levels. You can easily try the doubling effect at home: define a working CSV Reader and comment out one of the addField: commands before you use the NeoCSVReader to parse a CSV file. Say your CSV file has 3 lines with 4 columns each. If you remove one of the fieldAccessors, an #upToEnd will yoield an Array of 6 objects rather than 3. I haven't found the reason for the cases where this leads to an endless loop, but at least this one is clear... I *guess* this is due to the way #readEndOfLine is implemented. It seems to not peek forward to the end of the line. I have the gut feeling #peekChar should peek instead of reading the #next character form the input Stream, but #peekChar has too many senders to just go ahead and mess with it ;-) So I wonder if there are any tried approaches to this problem. One thing I might do is not use #upToEnd, but read each line using PositionableStream>>#nextLine and first check each line if the number of separators matches the number of fieldAccessors minus 1 (and go through the hoops of handling separators in quoted fields and such...). Only if that test succeeds, I would then hand a Stream with the whole line to the reader and do a #next. This will, however, mean a lot of extra cycles for large files. Of course I could do this only for some lines, maybe just the first one. Whatever. But somehow I have the feeling I should get an exception telling me the line is not compatible to the Reader's definition or such. Or #readAtEndOrEndOfLine should just walk the line to the end and ignore the rest of the line, returnong an incomplete object.... Maybe I am just missing the right setting or switch? What best practices did you guys come up with for such problems? Thanks in advance, Joachim

Jan. 4, 2021

Re: is there a better way

by Richard O'Keefe

What do you want the code to do? Have you profiled the code to see where the time is going? A quick look at the code shows - Paintings does one web get - each Painting does two more web gets ! and the first of those seems to be pretty pointless, as it refetches an object that Paintings already fetched and just looked at. On Sun, 3 Jan 2021 at 01:16, Roelof Wobben via Pharo-users < pharo-users(a)lists.pharo.org> wrote: > Hello, > > I have now this code : https://github.com/RoelofWobben/Rijksmuseam > > but it seems to be slow. > > Can anyone help me with a way I can use a sort of cache so the page > looks first at the cache if a image is there . > If so, take the image from there , if not , ask the api for the url of > the image. > > Roelof >

Jan. 3, 2021

Jump to page: