Hi Hernan I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is self longAt: 33 bigEndian: false -> 577660194 where it should have answered an Integer between 0 and 31 shapeType "position 32, Integer between 0 and 31, Little Endian" ^shpBytes longAt: 33 bigEndian: false No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok... Hans You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for" I wrote this script to reproduce: | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef | " Download Shapefile resources " urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix. #('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ]. Once downloaded: | urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. " Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp. Maybe it's related with the fact they are encoded as koi8r? I checked with Zinc downloading first from the browser: | shpFile | shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ]. Cheers, Hernán
Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems. ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you wonât see much (243 hardly visible dots). With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps... Hans From: Baveco, Hans <hans.baveco@wur.nl> Sent: vrijdag 5 juni 2020 12:44 To: Any question about pharo is welcome <pharo-users@lists.pharo.org> Subject: [Pharo-users] shape file reading Hi Hernan I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is self longAt: 33 bigEndian: false -> 577660194 where it should have answered an Integer between 0 and 31 shapeType "position 32, Integer between 0 and 31, Little Endian" ^shpBytes longAt: 33 bigEndian: false No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok... Hans You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for" I wrote this script to reproduce: | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef | " Download Shapefile resources " urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix. #('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ]. Once downloaded: | urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. " Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp. Maybe it's related with the fact they are encoded as koi8r? I checked with Zinc downloading first from the browser: | shpFile | shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ]. Cheers, Hernán
Hi Hans, Roassal has support for loading OSM raw data and OSM tiles (map backgrounds) (WGS84 projection). See https://www.youtube.com/watch?v=6zoGIsQaGUo&t=429s. We've also experimented with animations as well: https://www.youtube.com/watch?v=sMB7DG-lbiU; we've also tried in the past to work out of GRASS-produced rasters for complex geographic analysis, such as runoff flooding, projected on osm tiles. Regards, Thierry Le ven. 5 juin 2020 à 14:27, Baveco, Hans <hans.baveco@wur.nl> a écrit :
Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.
ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you wonât see much (243 hardly visible dots).
With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...
Hans
From: Baveco, Hans <hans.baveco@wur.nl> Sent: vrijdag 5 juni 2020 12:44 To: Any question about pharo is welcome <pharo-users@lists.pharo.org> Subject: [Pharo-users] shape file reading
Hi Hernan
I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is
self longAt: 33 bigEndian: false -> 577660194
where it should have answered an Integer between 0 and 31
shapeType
"position 32, Integer between 0 and 31, Little Endian"
^shpBytes longAt: 33 bigEndian: false
No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok...
Hans
You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"
I wrote this script to reproduce:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].
Once downloaded:
| urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'.
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp.
Maybe it's related with the fact they are encoded as koi8r?
I checked with Zinc downloading first from the browser:
| shpFile | shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ].
Cheers,
Hernán
El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<hans.baveco@wur.nl>) escribió:
Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.
ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you wonât see much (243 hardly visible dots).
I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely: | shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef | " Download Shapefile resources " urlRoot := ' https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix. #('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext | ZnClient new url: (urlPath , ext) asZnUrl; queryAt: 'raw' put: 'true'; numberOfRetries: 2; enforceHttpSuccess: true; downloadTo: urlESRIFilePrefix , ext; get ]. " Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp. " List data fields " shpE dataFields inspect. " List all shape records " shpE shapeRecords inspect. " Set the current attribute " shpE attribute: 'NAME'. legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute. shpE legend: legend. shpE displayMorphic.
With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...
Hans
*From:* Baveco, Hans <hans.baveco@wur.nl> *Sent:* vrijdag 5 juni 2020 12:44 *To:* Any question about pharo is welcome <pharo-users@lists.pharo.org> *Subject:* [Pharo-users] shape file reading
Hi Hernan
I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is
self longAt: 33 bigEndian: false -> 577660194
where it should have answered an Integer between 0 and 31
shapeType
"position 32, Integer between 0 and 31, Little Endian"
^shpBytes longAt: 33 bigEndian: false
No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok...
Hans
You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"
I wrote this script to reproduce:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := ' https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/ '. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].
Once downloaded:
| urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'.
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp.
Maybe it's related with the fact they are encoded as koi8r?
I checked with Zinc downloading first from the browser:
| shpFile |
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ].
Cheers,
Hernán
Please add this to the readme :)
On 5 Jun 2020, at 23:19, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<hans.baveco@wur.nl <mailto:hans.baveco@wur.nl>>) escribió: Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.
ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you wonât see much (243 hardly visible dots).
I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/ <https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/>'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext | ZnClient new url: (urlPath , ext) asZnUrl; queryAt: 'raw' put: 'true'; numberOfRetries: 2; enforceHttpSuccess: true; downloadTo: urlESRIFilePrefix , ext; get ].
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp. " List data fields " shpE dataFields inspect. " List all shape records " shpE shapeRecords inspect. " Set the current attribute " shpE attribute: 'NAME'.
legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute. shpE legend: legend.
shpE displayMorphic.
With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...
Hans
From: Baveco, Hans <hans.baveco@wur.nl <mailto:hans.baveco@wur.nl>> Sent: vrijdag 5 juni 2020 12:44 To: Any question about pharo is welcome <pharo-users@lists.pharo.org <mailto:pharo-users@lists.pharo.org>> Subject: [Pharo-users] shape file reading
Hi Hernan
I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is
self longAt: 33 bigEndian: false -> 577660194
where it should have answered an Integer between 0 and 31
shapeType
"position 32, Integer between 0 and 31, Little Endian"
^shpBytes longAt: 33 bigEndian: false
No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok...
Hans
You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"
I wrote this script to reproduce:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := 'https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/ <https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/>'. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].
Once downloaded:
| urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'.
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp.
Maybe it's related with the fact they are encoded as koi8r?
I checked with Zinc downloading first from the browser:
| shpFile | shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ].
Cheers,
Hernán
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
Done! Cheers Hernán El sáb., 6 jun. 2020 a las 3:17, Stéphane Ducasse (< stephane.ducasse@inria.fr>) escribió:
Please add this to the readme :)
On 5 Jun 2020, at 23:19, Hernán Morales Durand <hernan.morales@gmail.com> wrote:
El vie., 5 jun. 2020 a las 9:27, Baveco, Hans (<hans.baveco@wur.nl>) escribió:
Well, it appears that simple direct downloading does not download the real shapefile. Taking care to actually get a shapefile, the result loads without problems.
ne_110m_populated_places.shp is a PointShape with 243 shapes (points). If you display it on a Form (#displayMorphic) you wonât see much (243 hardly visible dots).
I checked and my previous download method was completely wrong. Now switching from ZnEasy to ZnClient to download and it works nicely:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := ' https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/ '. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp' '.cpg' '.prj') do: [ : ext | ZnClient new url: (urlPath , ext) asZnUrl; queryAt: 'raw' put: 'true'; numberOfRetries: 2; enforceHttpSuccess: true; downloadTo: urlESRIFilePrefix , ext; get ].
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp. " List data fields " shpE dataFields inspect. " List all shape records " shpE shapeRecords inspect. " Set the current attribute " shpE attribute: 'NAME'.
legend := ColorLegend mapValuesToRandom: shpE valuesOfCurrentAttribute. shpE legend: legend.
shpE displayMorphic.
With an application for vector graphics (Roassal?) that can read & display the vector data we would get much nicer maps...
Hans
*From:* Baveco, Hans <hans.baveco@wur.nl> *Sent:* vrijdag 5 juni 2020 12:44 *To:* Any question about pharo is welcome <pharo-users@lists.pharo.org> *Subject:* [Pharo-users] shape file reading
Hi Hernan
I downloaded this shapefile directly via the browser. Indeed it wonât load, the type of shape file returned is
self longAt: 33 bigEndian: false -> 577660194
where it should have answered an Integer between 0 and 31
shapeType
"position 32, Integer between 0 and 31, Little Endian"
^shpBytes longAt: 33 bigEndian: false
No idea what could be the cause â apparently the header of the file (the first 100 bytes) is not ok...
Hans
You mean some types are missing? Because I get this error when I try to read some SHP files: "Types > 20 are not yet accounted for"
I wrote this script to reproduce:
| shpE legend urlRoot urlESRIFilePrefix urlESRIFileShp urlPath response fileRef |
" Download Shapefile resources " urlRoot := ' https://github.com/nvkelso/natural-earth-vector/blob/master/110m_cultural/ '. urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'. urlPath := urlRoot , urlESRIFilePrefix.
#('.shx' '.dbf' '.shp') do: [ : ext | | url | url := (urlPath , ext , '?raw=true') asZnUrl. (response := ZnEasy get: url) isSuccess ifFalse: [ self error: 'Cannot download file' ]. fileRef := FileSystem disk workingDirectory / (urlESRIFilePrefix , ext). fileRef writeStreamDo: [ : stream | stream nextPutAll: ((ZnByteEncoder newForEncoding: 'koi8r') decodeBytes: response contents ) ] ].
Once downloaded:
| urlESRIFilePrefix urlESRIFileShp shpE | urlESRIFilePrefix := 'ne_110m_populated_places'. urlESRIFileShp := urlESRIFilePrefix , '.shp'.
" Load and display it in Morphic " shpE := ShapeEnsemble fromFile: urlESRIFileShp.
Maybe it's related with the fact they are encoded as koi8r?
I checked with Zinc downloading first from the browser:
| shpFile |
shpFile := 'ne_110m_admin_0_countries.shp' asFileReference. [ shpFile binaryReadStreamDo: [ : in | (ZnCharacterEncoder detectEncoding: in upToEnd) identifier ] ] on: ZnCharacterEncodingError do: [ nil ].
Cheers,
Hernán
-------------------------------------------- Stéphane Ducasse http://stephane.ducasse.free.fr / http://www.pharo.org 03 59 35 87 52 Assistant: Aurore Dalle FAX 03 59 57 78 50 TEL 03 59 35 86 16 S. Ducasse - Inria 40, avenue Halley, Parc Scientifique de la Haute Borne, Bât.A, Park Plaza Villeneuve d'Ascq 59650 France
participants (4)
-
Baveco, Hans -
Hernán Morales Durand -
Stéphane Ducasse -
Thierry Goubier