Difference between revisions of "Geonode data upload"

From stgo
Jump to: navigation, search
(Created page with ">> return to Cedeus IDE ---- == Problems I discovered when Uploading Data to GeoNode == === Spanish characters === Problem: Sometimes, after upload names of attributes...")
 
(→‎Loading Exiting Layers from GeoServer)
 
(21 intermediate revisions by one user not shown)
Line 2: Line 2:
 
----
 
----
  
−
== Problems I discovered when Uploading Data to GeoNode ==
+
== Loading Existing Layers from PostGIS ==
 +
 
 +
* Step 1: add the layer from PostGIS to GeoServer (using "add layer" option; however, PostGIS needs to be defined already as datastore)
 +
* Step 2: update a particular layer's info in GeoNode using ''"geonode updatelayers"''; e.g.
 +
*: <code>sudo geonode updatelayers --ignore-errors --filter=catastrobosquenativo_vii_maule</code>
 +
 
 +
== Loading Existing Image Layers from GeoServer ==
 +
* I had a bunch of images from Cediz (no mosaik)
 +
* I added these image first with the "import data" option in GeoServer (selecting the folder, creating a new workspace - like cediz-images - and then importing every image rather one-by-one by selecting and clicking on the "import" button) => do also check if they are visible in the layer preview, after import
 +
* Then I imported the layers from GeoServer into GeoNode with:
 +
<code>sudo geonode updatelayers --ignore-errors --workspace=cediz-images</code>
 +
: => it took some time though (I guess because of the thumbnail creation for some 1GB size images)
 +
 
 +
== Dealing with Problems I discovered when Uploading Data to GeoNode ==
 +
 
 +
=== Nothing happens when pressing Upload Button ===
 +
 
 +
'''Problem:''' Sometimes nothing happens when the shapefiles are selected and the upload button is pressed.
 +
 
 +
'''Solution:'''
 +
Check if the file name contains
 +
* spaces
 +
* spanish/latin characters or similar.
 +
If yes: rename the file to remove these characters.
 +
 
 +
=== File upload size limits ===
 +
 
 +
It may happen that the file to be uploaded to GeoNode hits the current size limit of 100 MB.
 +
 
 +
Its possible to change the size limit in ''/etc/geonode/local_settings.py'' with the entry <code>MAX_DOCUMENT_SIZE = 500 # MB</code>. Afterwards, restart Apache. However, its best to apply this larger limit only temporarily.
 +
 
 +
One 400MB shapefile, Uso de Suelo Aysen from MMA (with db = 750MB), I could not upload to GeoNode. Instead I copied it to the Geonode1204b VM and imported it there using:
 +
:<code>sudo geonode importlayers /home/myuser/xxx/</code>  (see GeoNode [http://geonode.readthedocs.org/en/latest/tutorials/admin/commands.html documentation])
 +
Before I actually changed the shapefile name to avoid latin characters.
 +
Afterwards I used "geonode updatelayers --ignore-errors --skip-geonode-registered"
  
 
=== Spanish characters ===
 
=== Spanish characters ===
  
−
Problem:  
+
'''Problem:'''
 
Sometimes, after upload names of attributes or attribute values are not correctly represented, i.e. <code>Categor�A</code>
 
Sometimes, after upload names of attributes or attribute values are not correctly represented, i.e. <code>Categor�A</code>
  
−
Solution:  
+
'''Solution:'''
 
This is an encoding problem. Before hitting the upload button in geonode, select the file encoding. In most cases it encoding is <code>Latin 1</code>.
 
This is an encoding problem. Before hitting the upload button in geonode, select the file encoding. In most cases it encoding is <code>Latin 1</code>.
  
 
=== Transformation error ===
 
=== Transformation error ===
  
−
Problem:
+
'''Problem:'''
 
The shapefile upload fails with an error. In particular I get:  
 
The shapefile upload fails with an error. In particular I get:  
 
  java.lang.RuntimeException: java.io.IOException: transform error
 
  java.lang.RuntimeException: java.io.IOException: transform error
  
 
and also (looking ''/usr/share/geoserver/data/logs/geoserver.log'') I got:
 
and also (looking ''/usr/share/geoserver/data/logs/geoserver.log'') I got:
−
  2015-03-11 15:47:05,991 WARN [geotools.jdbc] - Failed to use ST_Estimated_Extent, falling back on envelope aggregation
+
  Caused by: java.io.IOException: transform error
−
org.postgresql.util.PSQLException: ERROR: stats for "red_ferroviaria.the_geom" do not exist
+
        at org.geoserver.catalog.CatalogBuilder.getLatLonBounds(CatalogBuilder.java:525)
−
  Where: SQL function "st_estimated_extent" statement 2
+
        at org.geoserver.catalog.CatalogBuilder.doSetupBounds(CatalogBuilder.java:466)
 +
        at org.geoserver.catalog.CatalogBuilder.setupBounds(CatalogBuilder.java:450)
 +
        at org.geoserver.catalog.rest.DataStoreFileResource.handlePut(DataStoreFileResource.java:405)
 +
        ... 88 more
 +
Caused by: org.geotools.referencing.operation.projection.ProjectionException: Transformation doesn't convergence.
 +
        at org.geotools.referencing.operation.projection.MapProjection.inv_mlfn(MapProjection.java:1385)
 +
        at org.geotools.referencing.operation.projection.TransverseMercator.inverseTransformNormalized(TransverseMercator.java:217)
 +
 
 +
'''Reason:'''
 +
In some cases, there may be points/geometries with coordinates at 0/0 that cannot be transformed.
 +
 
 +
'''Solution:'''
 +
The problem is that there seems to be a misfit with the projection. Remember:
 +
* Chile's Easter Island is far off, so UTM 19S is perhaps not so good working as it is not in zone 19
 +
To solve this issue I re-saved the shapefile as WGS84 (epsg 4326) in geographic coordinates (and also I used UTF-8 character encoding, just to make sure). This new shapefile I was able to load in GeoNode 2.0
 +
 
 +
=== "quantize with one INF" Error ===
 +
 
 +
Exception: ('Failed to process accidentesgs2013_epsg4326', InvalidOperation('quantize with one INF',))
 +
 
 +
this error seems to occur when no bounds can be calculated ''after upload'' by GeoServer. Means, I got an infinite sign in one of the bound text boxes. When I recalculated or set the Lat/Lon bound manually it was ok then (at least for the ''accidentesgs2013'' layer) .
 +
 
 +
To get the layer into GeoNode I used <code>geonode updatelayers</code>.
 +
 
 +
== Adding WMS Layers from External Server (e.g. another GeoNode instance) ==
 +
 
 +
For example including a layer from CIGIDEN IDE:
 +
* check what is the WMS server gtCapabilities address
 +
* open GeoServer admin interface
 +
* create a new workspace (e.g. External-WMS)
 +
* add a new store of type "WMS"
 +
** put in the "''Capabilities URL''" field, the WMS address, e.g. "http://ide.cigiden.cl/geoserver/wms"
 +
* select the layer of the remote WMS that one wishes to publish (e.g. "riesgo_volc_villarrica")
 +
* use updatelayers command to update the store data to GeoNode:
 +
*: <code>sudo geonode updatelayers --ignore-errors --store=ide.CIGIDEN</code>
 +
 
 +
== Styling a Raster Layer in GeoServer ==
 +
 
 +
When uploading a raster via the GeoNode interface a new style (related to the file name) is created in GeoServer. This style can be edited.
 +
For instance, I used the following SLD for colouring the geotiff, result of a [http://cedeusdata.geosteiniger.cl/layers/geonode:accidents_stgo2000_2010_kde_h100m KDE analysis of traffic accidents]:
  
−
Solution:
+
<?xml version="1.0" encoding="UTF-8"?><sld:StyledLayerDescriptor xmlns="http://www.opengis.net/sld" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml" version="1.0.0">
−
The problem is that there seems to be a misfit with the projection (remember, Chile's Easter Island is far off, so UTM 19S is perhaps not so good working as it is not in zone 19). To solve this issue I re-saved the shapefile as WGS84 (epsg 4326) in geographic coordinates (and also I used UTF-8 character encoding, just to make sure). This new shapefile I was able to load in GeoNode 2.0
+
  <sld:NamedLayer>
 +
    <sld:Name>accidents_stgo2000_2010_kde_h100m</sld:Name>
 +
    <sld:UserStyle>
 +
      <sld:Name>accidents_stgo2000_2010_kde_h100m</sld:Name>
 +
      <sld:Title>accidents_stgo2000_2010_kde_h100m</sld:Title>
 +
      <sld:FeatureTypeStyle>
 +
        <sld:Name>name</sld:Name>
 +
        <sld:Rule>
 +
                    <sld:RasterSymbolizer>
 +
                        <sld:Geometry>
 +
                            <ogc:PropertyName>grid</ogc:PropertyName>
 +
                        </sld:Geometry>
 +
                        <sld:ColorMap>
 +
                            <sld:ColorMapEntry color="#000000" opacity="0.0" quantity="0.0"/>
 +
                            <sld:ColorMapEntry color="#FFCC00" opacity="0.5" quantity="0.01"/>
 +
                            <sld:ColorMapEntry color="#FF9900" opacity="0.5" quantity="10.0"/>                         
 +
                            <sld:ColorMapEntry color="#AA0000" opacity="0.8" quantity="100.0"/>
 +
                        </sld:ColorMap>
 +
                    </sld:RasterSymbolizer>
 +
        </sld:Rule>
 +
      </sld:FeatureTypeStyle>
 +
    </sld:UserStyle>
 +
  </sld:NamedLayer>
 +
</sld:StyledLayerDescriptor>

Latest revision as of 18:01, 11 March 2016

>> return to Cedeus IDE

Loading Existing Layers from PostGIS

  • Step 1: add the layer from PostGIS to GeoServer (using "add layer" option; however, PostGIS needs to be defined already as datastore)
  • Step 2: update a particular layer's info in GeoNode using "geonode updatelayers"; e.g.
    sudo geonode updatelayers --ignore-errors --filter=catastrobosquenativo_vii_maule

Loading Existing Image Layers from GeoServer

  • I had a bunch of images from Cediz (no mosaik)
  • I added these image first with the "import data" option in GeoServer (selecting the folder, creating a new workspace - like cediz-images - and then importing every image rather one-by-one by selecting and clicking on the "import" button) => do also check if they are visible in the layer preview, after import
  • Then I imported the layers from GeoServer into GeoNode with: 
sudo geonode updatelayers --ignore-errors --workspace=cediz-images
=> it took some time though (I guess because of the thumbnail creation for some 1GB size images)

Dealing with Problems I discovered when Uploading Data to GeoNode

Nothing happens when pressing Upload Button

Problem: Sometimes nothing happens when the shapefiles are selected and the upload button is pressed.

Solution: Check if the file name contains

  • spaces
  • spanish/latin characters or similar.

If yes: rename the file to remove these characters.

File upload size limits

It may happen that the file to be uploaded to GeoNode hits the current size limit of 100 MB.

Its possible to change the size limit in /etc/geonode/local_settings.py with the entry MAX_DOCUMENT_SIZE = 500 # MB. Afterwards, restart Apache. However, its best to apply this larger limit only temporarily.

One 400MB shapefile, Uso de Suelo Aysen from MMA (with db = 750MB), I could not upload to GeoNode. Instead I copied it to the Geonode1204b VM and imported it there using:

sudo geonode importlayers /home/myuser/xxx/ (see GeoNode documentation)

Before I actually changed the shapefile name to avoid latin characters. Afterwards I used "geonode updatelayers --ignore-errors --skip-geonode-registered"

Spanish characters

Problem: Sometimes, after upload names of attributes or attribute values are not correctly represented, i.e. Categor�A

Solution: This is an encoding problem. Before hitting the upload button in geonode, select the file encoding. In most cases it encoding is Latin 1.

Transformation error

Problem: The shapefile upload fails with an error. In particular I get: 

java.lang.RuntimeException: java.io.IOException: transform error

and also (looking /usr/share/geoserver/data/logs/geoserver.log) I got: 

Caused by: java.io.IOException: transform error
       at org.geoserver.catalog.CatalogBuilder.getLatLonBounds(CatalogBuilder.java:525)
       at org.geoserver.catalog.CatalogBuilder.doSetupBounds(CatalogBuilder.java:466)
       at org.geoserver.catalog.CatalogBuilder.setupBounds(CatalogBuilder.java:450)
       at org.geoserver.catalog.rest.DataStoreFileResource.handlePut(DataStoreFileResource.java:405)
       ... 88 more
Caused by: org.geotools.referencing.operation.projection.ProjectionException: Transformation doesn't convergence.
       at org.geotools.referencing.operation.projection.MapProjection.inv_mlfn(MapProjection.java:1385)
       at org.geotools.referencing.operation.projection.TransverseMercator.inverseTransformNormalized(TransverseMercator.java:217)

Reason: In some cases, there may be points/geometries with coordinates at 0/0 that cannot be transformed.

Solution: The problem is that there seems to be a misfit with the projection. Remember:

  • Chile's Easter Island is far off, so UTM 19S is perhaps not so good working as it is not in zone 19

To solve this issue I re-saved the shapefile as WGS84 (epsg 4326) in geographic coordinates (and also I used UTF-8 character encoding, just to make sure). This new shapefile I was able to load in GeoNode 2.0

"quantize with one INF" Error

Exception: ('Failed to process accidentesgs2013_epsg4326', InvalidOperation('quantize with one INF',))

this error seems to occur when no bounds can be calculated after upload by GeoServer. Means, I got an infinite sign in one of the bound text boxes. When I recalculated or set the Lat/Lon bound manually it was ok then (at least for the accidentesgs2013 layer) .

To get the layer into GeoNode I used geonode updatelayers.

Adding WMS Layers from External Server (e.g. another GeoNode instance)

For example including a layer from CIGIDEN IDE:

  • check what is the WMS server gtCapabilities address
  • open GeoServer admin interface
  • create a new workspace (e.g. External-WMS)
  • add a new store of type "WMS"
  • select the layer of the remote WMS that one wishes to publish (e.g. "riesgo_volc_villarrica")
  • use updatelayers command to update the store data to GeoNode:
    sudo geonode updatelayers --ignore-errors --store=ide.CIGIDEN

Styling a Raster Layer in GeoServer

When uploading a raster via the GeoNode interface a new style (related to the file name) is created in GeoServer. This style can be edited. For instance, I used the following SLD for colouring the geotiff, result of a KDE analysis of traffic accidents: 

<?xml version="1.0" encoding="UTF-8"?><sld:StyledLayerDescriptor xmlns="http://www.opengis.net/sld" xmlns:sld="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gml="http://www.opengis.net/gml" version="1.0.0">
 <sld:NamedLayer>
   <sld:Name>accidents_stgo2000_2010_kde_h100m</sld:Name>
   <sld:UserStyle>
     <sld:Name>accidents_stgo2000_2010_kde_h100m</sld:Name>
     <sld:Title>accidents_stgo2000_2010_kde_h100m</sld:Title>
     <sld:FeatureTypeStyle>
       <sld:Name>name</sld:Name>
       <sld:Rule>
                   <sld:RasterSymbolizer>
                       <sld:Geometry>
                           <ogc:PropertyName>grid</ogc:PropertyName>
                       </sld:Geometry>
                       <sld:ColorMap>
                           <sld:ColorMapEntry color="#000000" opacity="0.0" quantity="0.0"/>
                           <sld:ColorMapEntry color="#FFCC00" opacity="0.5" quantity="0.01"/>
                           <sld:ColorMapEntry color="#FF9900" opacity="0.5" quantity="10.0"/>                          
                           <sld:ColorMapEntry color="#AA0000" opacity="0.8" quantity="100.0"/>
                       </sld:ColorMap>
                   </sld:RasterSymbolizer>
       </sld:Rule>
     </sld:FeatureTypeStyle>
   </sld:UserStyle>
 </sld:NamedLayer>
</sld:StyledLayerDescriptor>
Retrieved from "http://wiki.geosteiniger.cl/mediawiki-1.22.7/index.php?title=Geonode_data_upload&oldid=1955"