Pharo-users
By thread
pharo-users@lists.pharo.org
By month
Messages by month
- ----- 2026 -----
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
October 2019
- 74 participants
- 329 messages
Re: [Pharo-users] How to zip a WideString
by Tomohiro Oda
Sven,
Yes, ByteArray>>zipped/unzipped are simple, neat and intuitive way of
zipping/unzipping binary data.
I also love the new idioms. They look clean and concise.
Best Regards,
---
tomo
2019å¹´10æ3æ¥(æ¨) 20:14 Sven Van Caekenberghe <sven(a)stfx.eu>:
>
> Actually, thinking about this a bit more, why not add #zipped #unzipped to ByteArray ?
>
>
> ByteArray>>#zipped
> "Return a GZIP compressed version of the receiver as a ByteArray"
>
> ^ ByteArray streamContents: [ :out |
> (GZipWriteStream on: out) nextPutAll: self; close ]
>
> ByteArray>>#unzipped
> "Assuming the receiver contains GZIP encoded data,
> return the decompressed data as a ByteArray"
>
> ^ (GZipReadStream on: self) upToEnd
>
>
> The original oneliner then becomes
>
> 'string' utf8Encoded zipped.
>
> and
>
> data unzipped utf8Decoded
>
> which is pretty clear, simple and intention-revealing, IMHO.
>
> > On 3 Oct 2019, at 13:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote:
> >
> > Hi Tomo,
> >
> > Indeed, I stand corrected, it does indeed seem possible to use the existing gzip classes to work from bytes to bytes, this works fine:
> >
> > data := ByteArray streamContents: [ :out | (GZipWriteStream on: out) nextPutAll: 'foo 10 â¬' utf8Encoded; close ].
> >
> > (GZipReadStream on: data) upToEnd utf8Decoded.
> >
> > Now regarding the encoding option, I am not so sure that is really necessary (though nice to have). Why would anyone use anything except UTF8 (today).
> >
> > Thanks again for the correction !
> >
> > Sven
> >
> >> On 3 Oct 2019, at 12:41, Tomohiro Oda <tomohiro.tomo.oda(a)gmail.com> wrote:
> >>
> >> Peter and Sven,
> >>
> >> zip API from string to string works fine except that aWideString
> >> zipped generates malformed zip string.
> >> I think it might be a good guidance to define
> >> String>>zippedWithEncoding: and ByteArray>>unzippedWithEncoding: .
> >> Such as
> >> String>>zippedWithEncoding: encoder
> >> zippedWithEncoding: encoder
> >> ^ ByteArray
> >> streamContents: [ :stream |
> >> | gzstream |
> >> gzstream := GZipWriteStream on: stream.
> >> encoder
> >> next: self size
> >> putAll: self
> >> startingAt: 1
> >> toStream: gzstream.
> >> gzstream close ]
> >>
> >> and ByteArray>>unzippedWithEncoding: encoder
> >> unzippedWithEncoding: encoder
> >> | byteStream |
> >> byteStream := GZipReadStream on: self.
> >> ^ String
> >> streamContents: [ :stream |
> >> [ byteStream atEnd ]
> >> whileFalse: [ stream nextPut: (encoder nextFromStream:
> >> byteStream) ] ]
> >>
> >> Then, you can write something like
> >> zipped := yourLongWideString zippedWithEncoding: ZnCharacterEncoder utf8.
> >> unzipped := zipped unzippedWithEncoding: ZnCharacterEncoder utf8.
> >>
> >> This will not affect the existing zipped/unzipped API and you can
> >> handle other encodings.
> >> This zippedWithEncoding: generates a ByteArray, which is kind of
> >> conformant to the encoding API.
> >> And you don't have to create many intermediate byte arrays and byte strings.
> >>
> >> I hope this helps.
> >> ---
> >> tomo
> >>
> >> 2019/10/3(Thu) 18:56 Sven Van Caekenberghe <sven(a)stfx.eu>:
> >>>
> >>> Hi Peter,
> >>>
> >>> About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
> >>>
> >>> The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
> >>>
> >>> It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
> >>>
> >>> About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
> >>>
> >>> What DB and what sizes are we talking about ?
> >>>
> >>> Sven
> >>>
> >>>> On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
> >>>>
> >>>> Hello
> >>>>
> >>>> I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
> >>>>
> >>>> I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
> >>>>
> >>>> So my proposed solution is:
> >>>>
> >>>> For compression: myZipString := myWideString utf8Encoded asString zipped.
> >>>> For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
> >>>>
> >>>> As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
> >>>>
> >>>> Thanks in advance for any help.
> >>>>
> >>>> Peter Kenny
> >>>
> >>>
> >>
> >
>
>
Oct. 3, 2019
Re: [Pharo-users] How to zip a WideString
by Sven Van Caekenberghe
Actually, thinking about this a bit more, why not add #zipped #unzipped to ByteArray ?
ByteArray>>#zipped
"Return a GZIP compressed version of the receiver as a ByteArray"
^ ByteArray streamContents: [ :out |
(GZipWriteStream on: out) nextPutAll: self; close ]
ByteArray>>#unzipped
"Assuming the receiver contains GZIP encoded data,
return the decompressed data as a ByteArray"
^ (GZipReadStream on: self) upToEnd
The original oneliner then becomes
'string' utf8Encoded zipped.
and
data unzipped utf8Decoded
which is pretty clear, simple and intention-revealing, IMHO.
> On 3 Oct 2019, at 13:04, Sven Van Caekenberghe <sven(a)stfx.eu> wrote:
>
> Hi Tomo,
>
> Indeed, I stand corrected, it does indeed seem possible to use the existing gzip classes to work from bytes to bytes, this works fine:
>
> data := ByteArray streamContents: [ :out | (GZipWriteStream on: out) nextPutAll: 'foo 10 â¬' utf8Encoded; close ].
>
> (GZipReadStream on: data) upToEnd utf8Decoded.
>
> Now regarding the encoding option, I am not so sure that is really necessary (though nice to have). Why would anyone use anything except UTF8 (today).
>
> Thanks again for the correction !
>
> Sven
>
>> On 3 Oct 2019, at 12:41, Tomohiro Oda <tomohiro.tomo.oda(a)gmail.com> wrote:
>>
>> Peter and Sven,
>>
>> zip API from string to string works fine except that aWideString
>> zipped generates malformed zip string.
>> I think it might be a good guidance to define
>> String>>zippedWithEncoding: and ByteArray>>unzippedWithEncoding: .
>> Such as
>> String>>zippedWithEncoding: encoder
>> zippedWithEncoding: encoder
>> ^ ByteArray
>> streamContents: [ :stream |
>> | gzstream |
>> gzstream := GZipWriteStream on: stream.
>> encoder
>> next: self size
>> putAll: self
>> startingAt: 1
>> toStream: gzstream.
>> gzstream close ]
>>
>> and ByteArray>>unzippedWithEncoding: encoder
>> unzippedWithEncoding: encoder
>> | byteStream |
>> byteStream := GZipReadStream on: self.
>> ^ String
>> streamContents: [ :stream |
>> [ byteStream atEnd ]
>> whileFalse: [ stream nextPut: (encoder nextFromStream:
>> byteStream) ] ]
>>
>> Then, you can write something like
>> zipped := yourLongWideString zippedWithEncoding: ZnCharacterEncoder utf8.
>> unzipped := zipped unzippedWithEncoding: ZnCharacterEncoder utf8.
>>
>> This will not affect the existing zipped/unzipped API and you can
>> handle other encodings.
>> This zippedWithEncoding: generates a ByteArray, which is kind of
>> conformant to the encoding API.
>> And you don't have to create many intermediate byte arrays and byte strings.
>>
>> I hope this helps.
>> ---
>> tomo
>>
>> 2019/10/3(Thu) 18:56 Sven Van Caekenberghe <sven(a)stfx.eu>:
>>>
>>> Hi Peter,
>>>
>>> About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
>>>
>>> The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
>>>
>>> It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
>>>
>>> About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
>>>
>>> What DB and what sizes are we talking about ?
>>>
>>> Sven
>>>
>>>> On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
>>>>
>>>> Hello
>>>>
>>>> I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
>>>>
>>>> I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
>>>>
>>>> So my proposed solution is:
>>>>
>>>> For compression: myZipString := myWideString utf8Encoded asString zipped.
>>>> For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
>>>>
>>>> As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
>>>>
>>>> Thanks in advance for any help.
>>>>
>>>> Peter Kenny
>>>
>>>
>>
>
Oct. 3, 2019
Re: [Pharo-users] How to zip a WideString
by Sven Van Caekenberghe
Hi Tomo,
Indeed, I stand corrected, it does indeed seem possible to use the existing gzip classes to work from bytes to bytes, this works fine:
data := ByteArray streamContents: [ :out | (GZipWriteStream on: out) nextPutAll: 'foo 10 â¬' utf8Encoded; close ].
(GZipReadStream on: data) upToEnd utf8Decoded.
Now regarding the encoding option, I am not so sure that is really necessary (though nice to have). Why would anyone use anything except UTF8 (today).
Thanks again for the correction !
Sven
> On 3 Oct 2019, at 12:41, Tomohiro Oda <tomohiro.tomo.oda(a)gmail.com> wrote:
>
> Peter and Sven,
>
> zip API from string to string works fine except that aWideString
> zipped generates malformed zip string.
> I think it might be a good guidance to define
> String>>zippedWithEncoding: and ByteArray>>unzippedWithEncoding: .
> Such as
> String>>zippedWithEncoding: encoder
> zippedWithEncoding: encoder
> ^ ByteArray
> streamContents: [ :stream |
> | gzstream |
> gzstream := GZipWriteStream on: stream.
> encoder
> next: self size
> putAll: self
> startingAt: 1
> toStream: gzstream.
> gzstream close ]
>
> and ByteArray>>unzippedWithEncoding: encoder
> unzippedWithEncoding: encoder
> | byteStream |
> byteStream := GZipReadStream on: self.
> ^ String
> streamContents: [ :stream |
> [ byteStream atEnd ]
> whileFalse: [ stream nextPut: (encoder nextFromStream:
> byteStream) ] ]
>
> Then, you can write something like
> zipped := yourLongWideString zippedWithEncoding: ZnCharacterEncoder utf8.
> unzipped := zipped unzippedWithEncoding: ZnCharacterEncoder utf8.
>
> This will not affect the existing zipped/unzipped API and you can
> handle other encodings.
> This zippedWithEncoding: generates a ByteArray, which is kind of
> conformant to the encoding API.
> And you don't have to create many intermediate byte arrays and byte strings.
>
> I hope this helps.
> ---
> tomo
>
> 2019/10/3(Thu) 18:56 Sven Van Caekenberghe <sven(a)stfx.eu>:
>>
>> Hi Peter,
>>
>> About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
>>
>> The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
>>
>> It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
>>
>> About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
>>
>> What DB and what sizes are we talking about ?
>>
>> Sven
>>
>>> On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
>>>
>>> Hello
>>>
>>> I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
>>>
>>> I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
>>>
>>> So my proposed solution is:
>>>
>>> For compression: myZipString := myWideString utf8Encoded asString zipped.
>>> For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
>>>
>>> As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
>>>
>>> Thanks in advance for any help.
>>>
>>> Peter Kenny
>>
>>
>
Oct. 3, 2019
Re: [Pharo-users] How to zip a WideString
by PBKResearch
Hi Sven
The DB system I am using at the moment is OmniBase - despite Todd Blanchard's warning, I have decided to experiment with it. It has the advantage of being fully object based, though I am not yet using anything more elaborate than strings, dictionaries and arrays as the data types. One secondary advantage is that I still use Dolphin occasionally, and my version of Dolphin 6.1 comes with OmniBase built in. I have checked that an OmniBase DB built in Pharo can be read in Dolphin.
As to size, there is no problem with storing large strings in OmniBase, except for the amount of disk space occupied in total. I am looking far ahead - my toy development DB is only about 15MB, but if I get to where I want to be, it could be tens of GB. With modern machines this may not be a problem, but I thought there might come a time when I want to think about trade-offs between storage space and unzipping time. I had a few qualms when I looked inside my development DB; it seems that an OmniBase DB consists of a few smallish index files and one ginormous file called 'objects'. I am not sure how the OS will get on with a huge single file.
But all this is speculative at the moment. For now I shall continue with storing the strings unzipped (but utf8Encoded - thanks for such a neat facility), bearing in mind that if I need to save space later, my method as described will work.
Peter Kenny
-----Original Message-----
From: Pharo-users <pharo-users-bounces(a)lists.pharo.org> On Behalf Of Sven Van Caekenberghe
Sent: 03 October 2019 10:56
To: Any question about pharo is welcome <pharo-users(a)lists.pharo.org>
Subject: Re: [Pharo-users] How to zip a WideString
Hi Peter,
About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
What DB and what sizes are we talking about ?
Sven
> On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
>
> Hello
>
> I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
>
> I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
>
> So my proposed solution is:
>
> For compression: myZipString := myWideString utf8Encoded asString zipped.
> For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
>
> As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
>
> Thanks in advance for any help.
>
> Peter Kenny
Oct. 3, 2019
Re: [Pharo-users] How to zip a WideString
by Tomohiro Oda
Peter and Sven,
zip API from string to string works fine except that aWideString
zipped generates malformed zip string.
I think it might be a good guidance to define
String>>zippedWithEncoding: and ByteArray>>unzippedWithEncoding: .
Such as
String>>zippedWithEncoding: encoder
zippedWithEncoding: encoder
^ ByteArray
streamContents: [ :stream |
| gzstream |
gzstream := GZipWriteStream on: stream.
encoder
next: self size
putAll: self
startingAt: 1
toStream: gzstream.
gzstream close ]
and ByteArray>>unzippedWithEncoding: encoder
unzippedWithEncoding: encoder
| byteStream |
byteStream := GZipReadStream on: self.
^ String
streamContents: [ :stream |
[ byteStream atEnd ]
whileFalse: [ stream nextPut: (encoder nextFromStream:
byteStream) ] ]
Then, you can write something like
zipped := yourLongWideString zippedWithEncoding: ZnCharacterEncoder utf8.
unzipped := zipped unzippedWithEncoding: ZnCharacterEncoder utf8.
This will not affect the existing zipped/unzipped API and you can
handle other encodings.
This zippedWithEncoding: generates a ByteArray, which is kind of
conformant to the encoding API.
And you don't have to create many intermediate byte arrays and byte strings.
I hope this helps.
---
tomo
2019/10/3(Thu) 18:56 Sven Van Caekenberghe <sven(a)stfx.eu>:
>
> Hi Peter,
>
> About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
>
> The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
>
> It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
>
> About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
>
> What DB and what sizes are we talking about ?
>
> Sven
>
> > On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
> >
> > Hello
> >
> > I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
> >
> > I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
> >
> > So my proposed solution is:
> >
> > For compression: myZipString := myWideString utf8Encoded asString zipped.
> > For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
> >
> > As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
> >
> > Thanks in advance for any help.
> >
> > Peter Kenny
>
>
Oct. 3, 2019
Re: [Pharo-users] How to zip a WideString
by Sven Van Caekenberghe
Hi Peter,
About #zipped / #unzipped and the inflate / deflate classes: your observation is correct, these work from string to string, while clearly the compressed representation should be binary.
The contents (input, what is inside the compressed data) can be anything, it is not necessarily a string (it could be an image, so also something binary). Only the creator of the compressed data knows, you cannot assume to know in general.
It would be possible (and it would be very nice) to change this, however that will have serious impact on users (as the contract changes).
About your use case: why would your DB not be capable of storing large strings ? A good DB should be capable of storing any kind of string (full unicode) efficiently.
What DB and what sizes are we talking about ?
Sven
> On 3 Oct 2019, at 11:06, PBKResearch <peter(a)pbkresearch.co.uk> wrote:
>
> Hello
>
> I have a problem with text storage, to which I seem to have found a solution, but itâs a bit clumsy-looking. I would be grateful for confirmation that (a) there is no neater solution, (b) I can rely on this to work â I only know that it works in a few test cases.
>
> I need to store a large number of text strings in a database. To avoid the database files becoming too large, I am thinking of zipping the strings, or at least the less frequently accessed ones. Depending on the source, some of the strings will be instances of ByteString, some of WideString (because they contain characters not representable in one byte). Storing a WideString uncompressed seems to occupy 4 bytes per character, so I decided, before thinking of compression, to store the strings utf8Encoded, which yields a ByteArray. But zipped can only be applied to a String, not a ByteArray.
>
> So my proposed solution is:
>
> For compression: myZipString := myWideString utf8Encoded asString zipped.
> For decompression: myOutputString := myZipString unzipped asByteArray utf8Decoded.
>
> As I said, it works in all the cases I tried, whether WideString or not, but the chains of transformations look clunky somehow. Can anyone see a neater way of doing it? And can I rely on it working, especially when I am handling foreign texts with many multi-byte characters?
>
> Thanks in advance for any help.
>
> Peter Kenny
Oct. 3, 2019
Antlr4 + SmaCC
by Santiago Bragagnolo
Hi there!
I am trying to build a parser for visual basic. For doing so i found the
Antlr definition of VB on the microsoft site.
I was wondering if it would be possible to use SmaCC for generating this
parser. Does any have experience in adapting Antlr4 grammars to the SmaCC
format ?? Or any clue about where to start checking for a clue about it
would be helpful also!
Thanks in advance.
Santiago
Oct. 3, 2019
How to zip a WideString
by PBKResearch
Hello
I have a problem with text storage, to which I seem to have found a
solution, but it's a bit clumsy-looking. I would be grateful for
confirmation that (a) there is no neater solution, (b) I can rely on this to
work - I only know that it works in a few test cases.
I need to store a large number of text strings in a database. To avoid the
database files becoming too large, I am thinking of zipping the strings, or
at least the less frequently accessed ones. Depending on the source, some of
the strings will be instances of ByteString, some of WideString (because
they contain characters not representable in one byte). Storing a WideString
uncompressed seems to occupy 4 bytes per character, so I decided, before
thinking of compression, to store the strings utf8Encoded, which yields a
ByteArray. But zipped can only be applied to a String, not a ByteArray.
So my proposed solution is:
For compression: myZipString := myWideString utf8Encoded
asString zipped.
For decompression: myOutputString := myZipString unzipped
asByteArray utf8Decoded.
As I said, it works in all the cases I tried, whether WideString or not, but
the chains of transformations look clunky somehow. Can anyone see a neater
way of doing it? And can I rely on it working, especially when I am handling
foreign texts with many multi-byte characters?
Thanks in advance for any help.
Peter Kenny
Oct. 3, 2019
Re: [Pharo-users] Test method auto-generation in Calypso
by Hernán Morales Durand
Ok, any hint to split the code area in Calypso at least?
Salut!
Hernán
El mié., 25 sept. 2019 a las 1:06, Hernán Morales Durand (<
hernan.morales(a)gmail.com>) escribió:
> Hi,
>
> I have this idea of typing a method in a Browser and have another code
> area in the same browser where the method test is "automagically
> generated".
>
> This is, divide the current method pane in two: One for the method being
> written itself, and another for its test, where at least the method
> selector could be autogenerated by writing testXXXX
>
> Of course being able to jump to the method test would be super nice! (and
> with a keyboard shortcut). This reminds myself to write the method now
> instead of forget it forever.
>
> Any hints to make this happen in Calypso?
>
> Cheers,
>
> Hernán
>
>
Oct. 1, 2019