OSGeo Planet

Syndicate content
Planet OSGeo - http://planet.osgeo.org
Updated: 1 hour 55 min ago

BostonGIS: New in QGIS 3.2 Save Project to PostgreSQL

4 hours 42 min ago

We've got customers discovering PostGIS and GIS in general or migrating away from ArcGIS family of tools. When they ask, "How do I see my data?", we often point them at QGIS which is an open source GIS desktop with rich integration with PostGIS/PostgreSQL.

QGIS is something that is great for people who need to live in their GIS environment since it allows for easily laying on other datasources, web services and maps. The DBManager tool allows for more advanced querying (like writing Spatial SQL queries that take advantage of the 100s of functions PostGIS has to offer) , ability to import/export data, and create PostgreSQL views.

QGIS has this thing called Projects, which allow for defining map layers and the symbology associated with them. For example what colors do you color your roads, and any extra symbols, what field attributes do you overlay - street name etc. Projects are usually saved in files with a .qgs or .qgz extension. If you spent a lot of time styling these layers, chances are you want to share them with other people in your group. This can become challenging if your group is not connected via network share.

Continue reading "New in QGIS 3.2 Save Project to PostgreSQL"
Categories: OSGeo Planet

QGIS Blog: QGIS 3.2 Bonn is released!

Sun, 2018-06-24 19:57

We are pleased to announce the release of QGIS 3.2 ‘Bonn’. The city of Bonn was the location of our 16th developer meeting.


This is the second release in the 3.x series. It comes with tons of new features (see our visual changelog).

Packages and installers for all major platforms are available from downloads.qgis.org.

We would like to thank the developers, documenters, testers and all the many folks out there who volunteer their time and effort (or fund people to do so). From the QGIS community we hope you enjoy this release! If you wish to donate time, money or otherwise get involved in making QGIS more awesome, please wander along to qgis.org and lend a hand!

QGIS is supported by donors and sponsors. A current list of donors who have made financial contributions large and small to the project can be seen on our donors list. If you would like to become and official project sponsor, please visit our sponsorship page for details. Sponsoring QGIS helps us to fund our six monthly developer meetings, maintain project infrastructure and fund bug fixing efforts. A complete list of current sponsors is provided below – our very great thank you to all of our sponsors!

QGIS is Free software and you are under no obligation to pay anything to use it – in fact we want to encourage people far and wide to use it regardless of what your financial or social status is – we believe empowering people with spatial decision making tools will result in a better society for all of humanity.





Categories: OSGeo Planet

QGIS Blog: QGIS Grant Programme 2018 Results

Fri, 2018-06-22 19:03

We are extremely pleased to announce the winning proposals for our 2018 QGIS.ORG grant programme. Funding for the programme was sourced by you, our project donors and sponsorsNote: For more context surrounding our grant programme, please see:

The QGIS.ORG Grant Programme aims to support work from our community that would typically not be funded by client/contractor agreements, and that contributes to the broadest possible swathe of our community by providing cross-cutting, foundational improvements to the QGIS Project.

Voting to select the successful projects was carried out by our QGIS Voting Members. Each voting member was allowed to select up to 6 of the 14 submitted proposals by means of a ranked selection form. The full list of votes are available here (on the first sheet). The second sheet contains the calculations used to determine the winner (for full transparency). The table below summarizes the voting tallies for the proposals:


A couple of extra notes about the voting process:

  • The PSC has an ongoing program to fund documentation so elected to fund the QGIS Training Manual update even if this increases the total funded amount beyond the initial budget.
  • Although the budget for the grant programme was €25,000, the total amount for the winning proposals is €35,500. This increase is possible thanks to the generous support by our donors and sponsors this year.
  • Voting was carried out based on the technical merits of the proposals and the competency of the applicants to execute on these proposals.
  • No restrictions were in place in terms of how many proposals could be submitted per person / organization, or how many proposals could be awarded to each proposing person / organization.
  • Voting was ‘blind’ (voters could not see the existing votes that had been placed).

Of the 45 voting members, 29 registered their votes 17 community representatives and 12 user group representatives.

On behalf of the QGIS.ORG project, I would like to thank everyone who submitted proposals for this call!

A number of interesting and useful proposal didn’t make it because of our limited budget; we encourage organizations to pick up one of their choice and sponsor it.

Categories: OSGeo Planet

QGIS Polska: blog:i_spotkania_uzytkownikow_qgis_juz_za_nami

Fri, 2018-06-22 18:07
Pomysł na zorganizowanie tego spotkania kiełkował od lat, ale zawsze brakowało czasu na jego realizację. Główny problem polegał na określeniu docelowej grupy uczestników oraz formuły. Wiemy oczywiście, jak szerokie jest grono użytkowników programu QGIS, ale ta wiedza wcale nie ułatwiała zadania. Ostatecznie po przeprowadzeniu kilku ankiet na polskim forum QGIS postanowiliśmy zaserwować wszystkiego po trochu. Potrzebni byli tylko prelegenci, miejsce i termin. Zdecydowaliśmy się na 19 czerwca 2018…
Categories: OSGeo Planet

QGIS Polska: blog:i_spotkania_uzytkownikow_qgis_juz_za_nami

Fri, 2018-06-22 18:07
Pomysł na zorganizowanie tego spotkania kiełkował od lat, ale zawsze brakowało czasu na jego realizację. Główny problem polegał na określeniu docelowej grupy uczestników oraz formuły. Wiemy oczywiście, jak szerokie jest grono użytkowników programu QGIS, ale ta wiedza wcale nie ułatwiała zadania. Ostatecznie po przeprowadzeniu kilku ankiet na polskim forum QGIS postanowiliśmy zaserwować wszystkiego po trochu. Potrzebni byli tylko prelegenci, miejsce i termin. Zdecydowaliśmy się na 19 czerwca 2018…
Categories: OSGeo Planet

Marco Bernasocchi: Using Threads in PyQGIS3

Fri, 2018-06-22 11:30
While porting a plugin to QGIS3 I decided to also move all it’s threading infrastructure to QgsTasks. Here three possible variants to implement this. the first uses the static method QgsTask.fromFunction and is simpler to use. A great quick solution.… See more ›
Categories: OSGeo Planet

Paul Ramsey: PostGIS Polygon Splitting

Thu, 2018-06-21 20:00

One of the joys of geospatial processing is the variety of tools in the tool box, and the ways that putting them together can yield surprising results. I have been in the guts of PostGIS for so long that I tend to think in terms of primitives: either there’s a function that does what you want or there isn’t. I’m too quick to ignore the power of combining the parts that we already have.

A community member on the users list asked (paraphrased): “is there a way to split a polygon into sub-polygons of more-or-less equal areas?”

I didn’t see the question, which is lucky, because I would have said: “No, you’re SOL, we don’t have a good way to solve that problem.” (An exact algorithm showed up in the Twitter thread about this solution, and maybe I should implement that.)

PostGIS developer Darafei Praliaskouski did answer, and provided a working solution that is absolutely brilliant in combining the parts of the PostGIS toolkit to solve a pretty tricky problem. He said:

The way I see it, for any kind of polygon:

  • Convert a polygon to a set of points proportional to the area by ST_GeneratePoints (the more points, the more beautiful it will be, guess 1000 is ok);
  • Decide how many parts you’d like to split into, (ST_Area(geom)/max_area), let it be K;
  • Take KMeans of the point cloud with K clusters;
  • For each cluster, take a ST_Centroid(ST_Collect(point));
  • Feed these centroids into ST_VoronoiPolygons, that will get you a mask for each part of polygon;
  • ST_Intersection of original polygon and each cell of Voronoi polygons will get you a good split of your polygon into K parts.

Let’s take it one step at a time to see how it works.

We’ll use Peru as the example polygon, it’s got a nice concavity to it which makes it a little trickier than an average shape.

CREATE TABLE peru AS SELECT * FROM countries WHERE name = 'Peru'

Original Polygon (Petu)

Now create a point field that fills the polygon. On average, each randomly placed point ends up “occupying” an equal area within the polygon.

CREATE TABLE peru_pts AS SELECT (ST_Dump(ST_GeneratePoints(geom, 2000))).geom AS geom FROM peru WHERE name = 'Peru'

2000 Random Points

Now, cluster the point field, setting the number of clusters to the number of pieces you want the polygon divided into. Visually, you can now see the divisions in the polygon! But, we still need to get actual lines to represent those divisions.

CREATE TABLE peru_pts_clustered AS SELECT geom, ST_ClusterKMmeans(geom, 10) over () AS cluster FROM peru_pts;

10 Clusters

Using a point field and K-means clustering to get the split areas was inspired enough. The steps to get actual polygons are equally inspired.

First, calculate the centroid of each point cluster, which will be the center of mass for each cluster.

CREATE TABLE peru_centers AS SELECT cluster, ST_Centroid(ST_collect(geom)) AS geom FROM peru_pts_clustered GROUP BY cluster;

Centroids of Clusters

Now, use a voronoi diagram to get actual dividing edges between the cluster centroids, which end up closely matching the places where the clusters divide!

CREATE TABLE peru_voronoi AS SELECT (ST_Dump(ST_VoronoiPolygons(ST_collect(geom)))).geom AS geom FROM peru_centers;

Voronoi of Centrois

Finally, intersect the voronoi areas with the original polygon to get final output polygons that incorporate both the outer edges of the polgyon and the voronoi dividing lines.

CREATE TABLE peru_divided AS SELECT ST_Intersection(a.geom, b.geom) AS geom FROM peru a CROSS JOIN peru_voronoi b;

Intersection with Original Polygon


Clustering a point field to get mostly equal areas, and then using the voronoi to extract actual dividing lines are wonderful insights into spatial processing. The final picture of all the components of the calculation is also beautiful.

All the Components Together

I’m not 100% sure, but it might be possible to use Darafei’s technique for even more interesting subdivisions, like “map of the USA subdivided into areas of equal GDP”, or “map of New York subdivided into areas of equal population” by generating the initial point field using an economic or demographic weighting.

Categories: OSGeo Planet

gvSIG Team: 5as Jornadas gvSIG Uruguay: Información Geográfica en un ámbito abierto

Thu, 2018-06-21 17:21

Los próximos días 18 y 19 de octubre se celebrarán las 5as Jornadas gvSIG Uruguay y 3as Jornadas de Tecnologías Libres de Información Geográfica y Datos Abiertos en Montevideo (Uruguay), bajo el lema ‘Información Geográfica en un ámbito abierto’.

Desde ahora está abierto el periodo de recepción de resúmenes, los cuales pueden enviarse a la dirección de correo jornadas.uruguay@gvsig.org siguiendo la plantilla facilitada en el apartado Comunicaciones de la web del evento, donde pueden consultarse también las normas para el envío. Los tipos de comunicación admitidos son ponencia y póster.

Durante las jornadas se entregarán los premios del Concurso de Uso de Tecnologías Libres de Información Geográfica 2018, organizado por GeoForAll Iberoamérica y OSGeo. para el cual los usuarios de cualquier Tecnología Libre de Información Geográfica pueden enviar ya sus trabajos.

Las jornadas serán gratuitas, y el periodo de inscripción se abrirá el 24 de septiembre.

Así mismo, cualquier entidad interesada en colaborar con las jornadas, puede hacerlo de varias formas, que incluyen desde una aportación económica hasta el aporte de recursos que de forma equivalente cubran las necesidades de apoyo detectado por el comité organizador. Toda la información relacionada con ello está disponible en la web de las jornadas.

Categories: OSGeo Planet

GIS for Thought: QGIS OpenStreetMap Scales

Wed, 2018-06-20 09:00

Save as a text file ending in .xml like qgis_scales.xml

These are the scales OpenStreetMap tiles are rendered in for 96 dpi, so the map will look sharp on most monitors.

The xml file can then be loaded into the project from:

Project> Project Properties…> General> Project scales

<qgsScales version="1.0"> <scale value="1:554678932"/> <scale value="1:277339466"/> <scale value="1:138669733"/> <scale value="1:69334866"/> <scale value="1:34667433"/> <scale value="1:17333716"/> <scale value="1:8666858"/> <scale value="1:4333429"/> <scale value="1:2166714"/> <scale value="1:1083357"/> <scale value="1:541678"/> <scale value="1:270839"/> <scale value="1:135419"/> <scale value="1:67709"/> <scale value="1:33854"/> <scale value="1:16927"/> <scale value="1:8463"/> <scale value="1:4231"/> <scale value="1:2115"/> </qgsScales>


1,000,000 (QGIS default):

1,083,357 (OSM wiki):

1,155,584 (From: 3liz):

Scales from:
OSM wiki

Categories: OSGeo Planet

GeoServer Team: GeoServer 2.12.4 Release

Wed, 2018-06-20 08:44

We are happy to announce the release of GeoServer 2.12.4. Downloads are available (zipwar, and exe) along with docs and extensions.

This is a maintenance release and a recommend update production systems. This release is made in conjunction with GeoTools 18.4.

Highlights of this release are featured below, for more information please see the release notes (,2.12.2, | 2.12-RC1 | 2.12-beta).

  • Add forceLabels=on in the style editor map legend to help users,
  • Remove language warnings during Windows setup compilation and remove ‘work’ folder when uninstalling on Windows
  • Move MongoDB community module to supported status
Bug Fixes
  • Response time of WMS 1.3.0 significantly higher than vs WMS 1.x.x on systems whose axis in north/east order
  • Exception with NULL values with AggregateProcess
  • Style with Interpolate function causes NullPointerException on GetLegendGraphic
  • WFS with startIndex doesn’t return some results
  • Vector identifying feature info uses an undocumented system variable to set the default search area
  • Removing extensions with own configuration bits may cause GeoServer not to start up anymore
  • Windows Installation issue – upgrading GeoServer results in corrupt data_dir
  • Class java.util.Map$Entry is not whitelisted for XML parsing.
  • Add WMS GetMap and GetFeatureInfo tests for App-Schema MongoDB integration
  • CatalogRepository cannot find a store by name, if the store has just been added
  • WCS 1.0.0 generates wrong links in GetCapabilities
  • CatalogRepository should return a null on store not found, instead it throws a RuntimeException
  • Layer page will only show up to 25 bands, regardless of the actual set of bands available
  • Undocumented GDAL 2.3.0 CSV output geometry column name change breaks WPSOgrTest
Security Updates

Please update your production instances of GeoServer to receive the latest security updates and fixes.

If you encounter a security vulnerability in GeoServer, or any other open source software, please take care to report the issue in a responsible fashion.

About GeoServer 2.12 Series

Additional information on the 2.12 series:

Categories: OSGeo Planet

gvSIG Team: New mailing list for gvSIG Developers

Tue, 2018-06-19 11:05

A new mailing list for gvSIG Developers has been created, that replaces the previous one. This list will continue being the main contact point for English speaking developers to ask about any doubt or problem on gvSIG development (Java, Scripting…).

The previous mailing list was hosted in Joinup, but their support has been ended. Therefore, we have decided to migrate the mailing list to OSGeo.

The new mailing list is available here:

https://lists.osgeo.org/mailman/listinfo/gvsig-desktop-devel You can configure the mailing list to receive list traffic bunched in digests, or if you don’t want to receive the e-mails from the list you can choose that option at settings. Then you will be able to send the doubts to the list, and you can consult replies from


We also want to thank OSGeo for their offer to host the mailing list.

Categories: OSGeo Planet

GeoSolutions: Rilasciata nuova versione del profilo DCAT-AP IT per CKAN

Tue, 2018-06-19 08:43


Dear Reader,

We apologize in advance, but this post is for our italian readers (hence in Italian only) to announce that we have finalized a new version of the DCAT-AP_IT Metadata Profile leveraging on the CKAN Open Data product.

Siamo lieti di condividere con voi le ultime novità che caratterizzeranno la nuova versione dell’estensione CKAN per il supporto al profilo applicativo DCAT-AP_IT. Come forse molti di voi già sapranno il profilo per la documentazione dei dati delle pubbliche amministrazioni (DCAT-AP_IT), reso disponibile dall’Agenzia per l’Italia Digitale (AgID), è nato con l’obiettivo di armonizzare i metadati con cui vengono descritti i dataset pubblici, al fine di migliorarne la qualità e favorire il riuso delle informazioni.

La prima versione, rilasciata ufficialmente nel Febbraio del 2017 e disponibile gratuitamente con licenza AGPL v3.0, fu sostenuta in uno sforzo congiunto dalla Provincia di Bolzano/Sud Tirol e dalla Provincia di Trento e fornisce ancora oggi un insieme valido ed eterogeneo di funzionalità non solo per la creazione guidata di datasets, ma anche per l’integrazione di metadati provenienti da sorgenti esterne (CSW, RDF, JSON-LD) in conformità al Profilo Applicativo. Sviluppata con scrupolosa attenzione alla stabilità delle sue caratteristiche funzionali, l’estensione ckanext-dcatapit, disponibile su una repository dedicata sotto il nostro account GitHub, nasce garantendo la più alta compatibilità possibile con le altre estensioni che spesso sono presenti nelle piattaforme CKAN. Anche gli aspetti legati al multilinguismo e la localizzazione dell’interfaccia sono stati affrontati e resi disponibili per garantire la massima usabilità da parte di quelle realtà che li necessitano, come per esempio le Provincie di Bolzano/Sud Tirol e Trento: l’estensione fornisce i propri files di localizzazione che aiutano a snellire eventuali personalizzazioni in questi termini, mentre l’estensione ckanext-multilang fornisce supporto per il multilinguismo dei contenuti presenti nel catalogo (dataset, organizzazioni, gruppi, risorse e altro).

Ad oggi non pochi sono i portali open data italiani che utilizzano questa estensione e tra questi si annoverano sicuramente:

  • Il portale OpenData della Provincia di Bolzano/Sud Tirol
  • Il portale OpenData del Trentino
  • L’infrastruttura federata OpenDataNetwork per il capofila Città Metropolitana di Firenze, che raccoglie e distribuisce i dati di vari enti toscani tra cui: Città Metropolitana di Firenze, Provincia di Prato, Provincia di Pistoia ed Autorità di Bacino dell’Arno.

Ma anche molti altri tra cui:

Grazie all’interesse mostrato dall’Agenzia per l’Italia Digitale (AgID) riguardo alle potenzialità e alle caratteristiche proprie di questa estensione, hanno avuto inizio alla fine del 2017 gli sviluppi per la realizzazione di una nuova versione arricchita e migliorata.

Gli sforzi di AgID nel finanziare questo progetto hanno avuto l’obiettivo di creare un unico hub di raccolta nazionale per i dataset pubblici basato su CKAN e fornire quindi, in un unico punto di accesso, le principali informazioni sui dati aperti esposti dalle PA locali e centrali (si fa in particolar modo riferimento al progetto DAF e la sua componente Dataportal).

Un insieme eterogeneo di funzionalità e peculiari caratteristiche sono state introdotte nella nuova versione per garantire non solo una più completa adesione al Profilo Applicativo, ma anche per aiutare l’utente nella ricerca dei datasets con nuove funzioni di indicizzazione e raggruppamento dei dataset stessi per regione di provenienza. È stato aggiunto dunque il supporto alla cardinalità multipla per le proprietà che la richiedono (come per esempio gli identificativi del dataset,  temi e sottotemi, autori e altre) e le funzionalità di catalogo sono state arricchite per identificare il catalogo e l’organizzazione di origine dei dataset  harvestati. In aggiunta, nuove facets saranno disponibili per filtrare i datasets per catalogo di origine, regioni e sottotemi.

[caption id="attachment_4152" align="aligncenter" width="800"]Nuove facet di ricerca disponibili nella schermata di ricerca Nuove facet di ricerca disponibili nella schermata di ricerca[/caption] Le caratteristiche proprie della form di creazione e modifica del dataset sono state migliorate offrendo delle mini guide di inserimento più dettagliate per l’utente, mentre il supporto al multilinguismo, offerto dall’estensione ckanext-multilang, è stato esteso anche ad altre proprietà, come per esempio rightsHolder, publisher, creator e conformsTo sia in harvesting che in serializzazione del dataset. [caption id="attachment_4153" align="aligncenter" width="800"]Scheda di dettaglio del dataset Scheda di dettaglio del dataset[/caption] La web form di creazione del dataset è stata inoltre ristrutturata attraverso un flusso di editing basato su macro ambiti di inserimento, per meglio indirizzare l’utente nella valorizzazione delle proprietà richieste dal Profilo. [caption id="attachment_4160" align="aligncenter" width="800"]Nuova form di modifica del dataset Nuova form di modifica del dataset[/caption] Anche i vocabolari controllati, primo tra tutti quello delle licenze, sono stati aggiornati e la nuova estensione ckanext-dcatapit metterà a disposizione un campo addizionale per la licenza a livello di risorsa del dataset. [caption id="attachment_4155" align="aligncenter" width="800"]Impostazione della licenza per la risorsa Impostazione della licenza per la risorsa[/caption]  

Anche il supporto al vocabolario controllato dei sottotemi, precedentemente mancante, è stato introdotto insieme al vocabolario controllato per la classificazione del territorio che consente l’associazione di una o più regioni italiane ad ogni organizzazione e facilitare quindi la ricerca dei datasets.

[caption id="attachment_4161" align="aligncenter" width="800"]Nuova form di modifica del dataset, selezione di temi e sottotemi Nuova form di modifica del dataset, selezione di temi e sottotemi[/caption]

Gli sforzi per la realizzazione della nuova versione si sono concentrati anche sul consolidare e accrescere le funzionalità di harvesting dei dataset con lo scopo sia di organizzare e catalogare al meglio i dataset raccolti ma anche di correggere, per quanto possibile, eventuali difformità nei dataset di origine (per esempio temi non conformi al Profilo Applicativo). Tra gli aspetti importanti riguardanti l’harvesting dei dataset troviamo anche i seguenti:

  • Migliorata la validazione dei tag in modo da gestire tag non conformi
  • Introdotto la mappatura delle licenze non conformi con quelle del vocabolario controllato aggiornato
  • Consolidamento delle funzionalità di harvesting già esistenti

Come ultimo punto, ma non per questo meno importante, lo sviluppo di una infrastruttura basata su Docker è stato messo a disposizione (attualmente ancora in fase di testing in vista del prossimo rilascio). Questo progetto, sviluppato in parallelo dal team di GeoSolutions, è disponibile sul GitHub Developers Italia e mette a disposizione tutto ciò di cui avete bisogno per ottenere rapidamente ed in pochi passi un’installazione completa di CKAN corredata dell’estensione ckanext-dcatapit.

Invitiamo tutti coloro che sono interessati a partecipare allo sforzo per lo sviluppo di questa estensione o che fossero interessati ad utilizzare questa estensione a seguire il nostro blog o iscriversi alla nostra newsletter; raccomandiamo di visionare anche i nostri pacchetti di supporto professionale GeoSolutions Enterprise Support Services nel caso si volesse usufruire di un supporto attento e qualificato per la messa in produzione di questa estensione. Allo stesso modo vi invitiamo a visionare le informazioni sugli altri nostri prodotti Open Source quali GeoServerMapstore, GeoNode e GeoNetwork.

The GeoSolutions team,
Categories: OSGeo Planet

GeoTools Team: GeoTools 18.4 released.

Tue, 2018-06-19 08:35
GeoTools 18.4 Released The GeoTools team is pleased to announce the release of GeoTools 18.4: geotools-18.4-bin.zip geotools-18.4-doc.zip geotools-18.4-userguide.zip geotools-18.4-project.zip maven repository Thanks to everyone who contributed to this release. This release is made in conjunction with GeoServer 2.12.4. This release is a maintenance release providing bug fixes and
Categories: OSGeo Planet

BostonGIS: Coming PostGIS 2.5 ST_OrientedEnvelope

Mon, 2018-06-18 21:42

PostGIS 2.5 is just around the corner. One of the new functions coming is the ST_OrientedEnvelop. This is something I've been waiting for for years. It is the true minimum bounding rectangle, as opposed to ST_Envelope which is an axis aligned bounding rectable.

Below is a pictorial view showing the difference between the two.

Continue reading "Coming PostGIS 2.5 ST_OrientedEnvelope"
Categories: OSGeo Planet

gvSIG Team: gvSIG en el XVIII Congreso Nacional TIG

Mon, 2018-06-18 17:09


La Asociación gvSIG, fiel a su compromiso de dar visibilidad a los proyectos de SIG Libre, colabora con el XVIII Congreso Nacional TIG que se realizará en Valencia entre los días 20 y 22 de junio.

Tendremos un stand donde estaremos encantados de hablar con cualquier institución o empresa interesada en implementar cualquiera de los productos de gvSIG, informaremos de las últimas novedades de la Suite gvSIG y tendremos algún que otro obsequio para las personas visitantes.

Además, impartiremos un taller para aprender scripting en gvSIG Desktop. Será el jueves 21 de 16:00 a 17:30.

Os facilitamos el enlace para ver el programa completo:



Categories: OSGeo Planet

Even Rouault: The barn is raising

Mon, 2018-06-18 11:40
Thanks to the support given by the sponsors of the GDAL SRS barn effort, I have been able to kick in the first works in the past weeks. The work up to now has been concentrated on the PROJ front.
The first step was to set a foundation of C++ classes that implement the ISO-19111 / OGC Topic 2 "Referencing by coordinates" standard. Actually I have anticipated the future adoption of the 18-005r1 2018 revision of the standard that takes into account the latest advances in the modelling of coordinate reference systems (in particular dynamic reference frames, geoid-based vertical coordinate reference systems, etc.), which will be reflected in the corresponding update of the WKT2:2018 standard and future updates of the EPSG dataset. If you are curious, you can skim through the resulting PROJ class hierarchy which is really close to the abstract specification (a number of those classes currenty lack a real implementation for now). With the agreement of the newly born PROJ project steering committee, I have opted for C++11 which offers a number of useful modern features to reduce boilerplate and concentrate on the interesting aspects of the work.
On the functional front, there is already support to read WKT1 (its GDAL variant for now) and WKT2 strings and build a subset of the before mentionned C++ objects. And conversely to dump those C++ objects as WKT1 and WKT2 strings. In particular you can import from WKT1 and export to WKT2, or the reverse (within the limitations of each format). So this abstract modelling (quite close to WKT2 of course) effectively serves its purpose to help being independant from the actual representation of the CRS. As I mentionned an early adoption of the OGC Topic 2 standard, similarly I've taken into account the future WKT2:2018 (OGC 18-010) standard that aligns well with the abstract specification. In the API, the user can select if he wants to export according to the currently adopted version WKT2:2015 (OGC 12-063r5), or with the future WKT2:2018 revision.
The result of those first steps can be followed in this pull request.
Another task that has been accomplished is the addition of the Google Test C++ testing framework to PROJ (thanks to Mateusz Loskot for his help with the CMake integration), so all those new features can be correctly tested locally and on all platforms supported by PROJ continuous integration setup.
There are many future steps to do just on the PROJ front :
  • implement remaining classes
  • code documentation
  • comprehensive support of projection methods (at least the set currently supported by GDAL)
  • import from and export to PROJ strings for CRS definitions and coordinate operations
  • use of the EPSG dataset
Categories: OSGeo Planet

Even Rouault: SQLite and POSIX advisory locks

Sat, 2018-06-16 13:47
Those past couple days, I was working on implementing multi-layer transaction support for GeoPackage datasources (for QGIS 3.4). Multi-layer transaction is an advanced functionality of QGIS (you have to enable it in project settings), initially implemented for PostgreSQL connections where several layers can be edited together so as to have atomic modifications when editing them. Modifications are automatically sent to the database, using SQL savepoints to implement undo/redo operations, instead of being queued in memory and committed at once when the user stops editing  the layer.
While debugging my work during development, I stumbled upon a heisenbug. From time to time, the two auxiliary files attached to a SQLite database opened in Write Ahead Logging (WAL) mode, suffixed -wal and -shm, would suddenly disappear, whereas the file was still being opened by QGIS. As those files are absolutely required, the consequence of this was that following operations on the database failed: new readers (in the QGIS process) would be denied opening the file, and QGIS could not commit any new change to it. When the file was closed, the file returned again in a proper state (which shows the robustness of sqlite). After some time, I realized that my issue arised exactly when I observed the database being edited by QGIS with an external ogrinfo on it (another way to reproduce the issue would be to open a second QGIS instance on the same file and close it). I indeed used ogrinfo to check that the state of the database was consistent during the editing operations. Okay, so instead of a random bug, I had now a perfectly reproducable bug. Half of the way to solve it, right ?
How come ogrinfo, which involves read-only operations, could cause those -wal and -shm files to disappear ? I had some recollection of code I had written in the OGR SQLite driver regarding this. When a dataset opened in read-only mode is closed by OGR, it checks if there's a -wal file still existing (which could happen if a database had not been cleanly closed, like a killed process), and if so, it re-opens it temporarily in update mode, does a dummy operation on it, and close it. If the ogrinfo process is the only one that had a connection on the database, libsqlite would remove the -wal and -shm files automatically (OGR does not directly remove the file, it relies on libsqlite wisdom to determine if they can be removed or not). But wait, in my above situation, ogrinfo was not the exclusive process operating on the database: QGIS was still editing it.... Would that be a bug in the venerable libsqlite ??? (spoiler: no)
I tried to reproduce the situation with replacing QGIS by a plain sqlite console opening the file, and doing a ogrinfo on it. No accidental removal of the -wal and -shm files. Okay, so what is the difference between QGIS and the sqlite console (beside QGIS having like one million extra line of code;-)). Well, QGIS doesn't directly use libsqlite3 to open GeoPackage databases, but uses the OGR GPKG driver. So instead of opening with QGIS or a sqlite3 console, what if I opened with the OGR GPKG driver ? Bingo, in that situation, I could also reproduce the issue. So something in OGR was the culprit. I will save you of the other details, but at the end it turned out that if OGR was opening itself a .gpkg file using standard file API, whereas libsqlite3 was opening it, chaos would result. This situation can happen since for example when opening a dataset, OGR has to open the underlying file to at least read its header and figure out which driver would handle it. So the sequence of operation is normally:1) the GDALOpenInfo class opens the file2) the OGR GeoPackage driver realizes this file is for it, and use the sqlite3_open() API to open it3) the GDALOpenInfo class closes the file it has opened in step 1 (libsqlite3 still manages its own file handle)
When modifying the above sequence, so that 3) is executed before 2), the bug would not appear. At that point, I had some recollection that sqlite3 used POSIX advisory locks to handle concurrent accesses, and that there were some issues with that POSIX API. Digging into the sqlite3.c source code revealed a very interesting 86 line long comment about how POSIX advisory locks are broken by design. The main brokenness are they are advisory and not compulsory of course, but as this is indicated in the name, one cannot really complain about that being a hidden feature. The most interesting finding was: """If you close a file descriptor that points to a file that has locks, all locks on that file that are owned by the current process are released.""" Bingo: that was just what OGR was doing.My above workaround (to make sure the file is closed before sqlite opens it and set its locks) was OK for a single opening of a file in a process. But what if the user wants to open a second connection on the same file (which arises easily in the QGIS context) ? The rather ugly solution I came off was that the OGR GPKG driver would warn the GDALOpenInfo not to try to open a given file while it was still opened by the driver and pass it the file header it would be supposed to find if it could open the file, so that the driver identification logic can still work. Those fixes are queued for GDAL 2.3.1, whose release candidate is planned next Friday.
  • never ever open (actually close) a SQLite database with regular file API while libsqlite3 is operating on it (in the same process)
  • POSIX advisory locks are awful.
Categories: OSGeo Planet

GIS for Thought: GAA Winners 1887 to 2017 GIFs

Wed, 2018-06-13 09:00

Some of the outputs from my Data Driven Cartography workshop at the 2nd OSGeo Ireland conference.

Gaelic Football:

Final frame:
GAA football


Final frame:
GAA hurling

Categories: OSGeo Planet

GRASS GIS: GRASS GIS 7.4.1 released

Tue, 2018-06-12 20:06
We are pleased to announce the GRASS GIS 7.4.1 release
Categories: OSGeo Planet

Blog 2 Engenheiros: Como utilizar Python no QGIS (PyQGIS) para automatizar processos?

Tue, 2018-06-12 07:01

Quando trabalhamos com Sistemas de Informações Geográficas (SIG), alguns trabalhos são bastante repetitivos, beirando à chatice.

Extrair, cortar, realizar analise, extrair, cortar, realizar analise. Quantas vezes você vai ficar repetindo esse processo? Por que não programar para realizar todo ele de uma vez só?

Já apresentamos como realizar esse tipo de procedimento utilizando o ArcGIS, e caso você queira mais detalhes sobre o que é Python, dê uma olhada na nossa postagem sobre como usar Python e a ferramenta Buffer no ArcGIS.

Nesta postagem, utilizaremos o QGIS 2.18 e Python para extrair o limite do município de Cocal do Sul (e de outros municípios), em seguida, iremos utilizar esse limite para recortar o mapa de solos do Estado de Santa Catarina.

Nos links abaixo, você poderá baixar os shapefiles que usaremos nesta postagem.

Como acessar o Python no QGIS (PyQGIS)?

Antes de adicionarmos qualquer shapefile, vamos abrir o terminal e o editor python do QGIS para escrever e executar nossos comandos.

Vá em Plugins e clique em Python Console (1). Provavelmente somente o Terminal será aberto, mas você pode habilitar o editor clicando em “Show Editor” (2), conforme figura abaixo.

Iniciando o módulo Python no QGIS 2.18.Iniciando o módulo Python no QGIS 2.18.

Com todas essas janelas abertas, vamos começar a programar na janela que foi aberta quando clicamos em “Show Editor”.

Carregando shapefile usando PyQGIS

Agora, vamos carregar algumas bibliotecas e vamos utilizar a função addVectorLayer() para adicionar nossos dois shapefiles.

#!/usr/bin/env python #coding: utf-8 # Carregando bibliotecas no Python from PyQt4.QtCore import * from PyQt4.QtGui import * from qgis.core import * from qgis.gui import * import processing import sys # Adicionando nossos arquivos shapefile lim_mun = "C:/Users/ferna/Desktop/PyQGIS/42MUE250GC_SIR.shp" solos_sc = "C:/Users/ferna/Desktop/PyQGIS/Solos_Santa_Catarina_250000_2004.shp" iface.addVectorLayer(lim_mun, "Limites Municipais", "ogr") iface.addVectorLayer(solos_sc, "Solos de SC", "ogr")

A função addVectorLayer recebe três parâmetros, são eles (1) o local onde foi salvo o shapefile; (2) nome da camada e (3) o identificador da fonte de dados, normalmente ogr.

Extraindo o limite municipal

Agora, com as camadas carregadas, iremos acessar os dados da camada “Limite Municipal” e por meio da ferramenta Extract by Attribute, iremos separar o limite do município de Cocal do Sul em um novo shapefile.

Iremos utilizar a função “processing” e dentro dela vamos especificar o algoritmo para extração de dado baseando-se em um atributo.

# Rodando algoritmo para extrair limites de Cocal do Sul limCocal = "C:/Users/ferna/Desktop/PyQGIS/limCocal.shp" processing.runalg('qgis:extractbyattribute', lim_mun, "NM_MUNICIP", 0, "COCAL DO SUL", limCocal)

Com o limite municipal de Cocal do Sul em mãos, já podemos rodar o nosso código seguinte para recortar os tipos de solos existentes dentro do município.

Cortando a partir de um polígono

Neste procedimento, utilizaremos a função Clip para recortar as classes de solo do município de Cocal do Sul.

Mas antes de executarmos esse código, nossos shapefiles estão em sistemas de projeção diferentes, isso pode ocasionar erros no momento do cruzamento dos mapas. Por isso, antes de rodar o código de recorte, vamos executar o qgis:reprojectlayer, para converter o shapefile do tipo de solo para SIRGAS2000.

# Convertendo os sistemas de coordenadas Solos_SIRGAS = "C:/Users/ferna/Desktop/PyQGIS/Solos_SIRGAS.shp" processing.runalg("qgis:reprojectlayer", solos_sc, "epsg:4674", Solos_SIRGAS) # Recortando nossa area de estudo solosCocal = "C:/Users/ferna/Desktop/PyQGIS/solosCocal.shp" processing.runalg('qgis:clip', Solos_SIRGAS, limCocal, solosCocal) iface.addVectorLayer(solosCocal, "Solos de Cocal do Sul", "ogr")

E dessa forma, conseguimos extrair os tipos de solos do município de Cocal do Sul.

Note que as funções Reproject Layer e Clip precisam dos seguintes argumentos:

  • Reproject Layer: (1) Arquivo de Entrada, (2) Sistema de Coordenadas Final, (3) Arquivo de Saída.
  • Clip: (1) Arquivo a ser cortado, (2) Limites do corte, (3) Arquivo de Saída.
Como realizar esse processo para varias cidades?

Para isso, vamos construir uma variável com os nosso municípios alvos (i.e. Ermo, Forquilhinha, Nova Veneza e Garopaba) e em seguida, criaremos um loop do tipo for para rodar todos eles, veja abaixo.

# Extraindo multiplos mapas de solo com um loop cidades = ["ERMO", "FORQUILHINHA", "NOVA VENEZA", "GAROPABA"] for n in cidades: limite = "C:/Users/ferna/Desktop/PyQGIS/lim" + n +".shp" processing.runalg('qgis:extractbyattribute', lim_mun, "NM_MUNICIP", 0, n, limite) solos = "C:/Users/ferna/Desktop/PyQGIS/solos" + n + ".shp" processing.runalg('qgis:clip', Solos_SIRGAS, limite, solos) iface.addVectorLayer(solos, "Solos de " + n , "ogr")

Note que após realizar esse processo, você poderá substituir a variável cidades pelos municípios que você deseja, desde que o nome esteja igual ao nome do município no shapefile com os limites municipais.

Obs.: Estou pesquisando como extrair os municípios que apresentam caracteres especiais em seus nomes (i.e. acentos, cedilhas), pois até agora, não consegui extrair o shape de solos deles. Até onde pesquisei, tem a ver com a codificação dos caracteres (UTF-8). Caso você tenha a resposta, deixe ela nos comentários que estaremos agradecendo e atualizando a postagem.

[Atualização em 18/06/2018]

Corrigindo o problemas de Encoding

Depois de pesquisar um pouco sobre como converter um shapefile para um Encoding diferente, conseguimos realizar o procedimento anterior para municípios com acentos.

O novo código tem uma parte para realizar essa correção, onde carregamos novamente o shapefile (função QgsVectorLayer()) e em seguida salvamos ele num novo formato (função QgsVectorFileWrite().writeAsVectorFormat())

Lembre-se também de colocar um “u” na frente dos textos que contém acentos, como fizemos na variável cidades.

Confira o código completo abaixo.

# -*- coding: utf-8 -*- # Carregando pacotes no Python import processing import sys import osgeo.ogr as ogr import osgeo.osr as osr # Carregando nossos arquivos shapefile lim_mun = "C:/Users/ferna/Desktop/PyQGIS/42MUE250GC_SIR.shp" solos_sc = "C:/Users/ferna/Desktop/PyQGIS/Solos_Santa_Catarina_250000_2004.shp" # Convertendo os sistemas de coordenadas Solos_SIRGAS = "C:/Users/ferna/Desktop/PyQGIS/Solos_SIRGAS.shp" processing.runalg("qgis:reprojectlayer", solos_sc, "epsg:4674", Solos_SIRGAS) # Corrigindo Encoding Camada = QgsVectorLayer(lim_mun, None, 'ogr') Camada.setProviderEncoding("utf8") Camada.dataProvider().setEncoding("utf8") lim_UTF8 = "C:/Users/ferna/Desktop/PyQGIS/lim_UTF8.shp" QgsVectorFileWriter.writeAsVectorFormat(Camada, lim_UTF8, "utf_8_encode", Camada.crs(), "ESRI Shapefile") # Extraindo vários itens com acentos cidades = [u"CRICIÚMA", u"MORRO DA FUMAÇA"] for n in cidades: limite = "C:/Users/ferna/Desktop/PyQGIS/lim" + n +".shp" processing.runalg('qgis:extractbyattribute', lim_UTF8, "NM_MUNICIP", 0, n, limite) solos = "C:/Users/ferna/Desktop/PyQGIS/solos" + n + ".shp" processing.runalg('qgis:clip', Solos_SIRGAS, limite, solos) iface.addVectorLayer(solos, "Solos de " + n , "ogr")

Caso tenha ficado com alguma dúvida, deixe ela nos comentários que estaremos respondendo assim que possível.

Categories: OSGeo Planet