OSGeo Planet

GeoSolutions: GeoSolutions presente ad ASITA 2018

OSGeo Planet - Thu, 2018-11-08 10:00

ASITA 2018

GeoSolutions sarà presente alla 22esima conferenza nazionale ASITA dal 27 al 29 Novembre 2018 presso l'hotel Four Points Sheraton a Bolzano con un proprio stand, dove vi aspettano per parlare di come poter soddisfare i vostri bisogni attraverso i nostri prodotti Open Source GeoServer, MapStore e GeoNode ed attraverso i nostri piani di supporto professionale e consulenza.   Durante la conferenze i tecnici di GeoSolutions prenderanno parte Martedì 27 Novembre dalle ore 14:30 alle ore 18:30 alla sessione parallela #2 "Infrastrutture di dati geografici e interoperabilità" con i seguenti interventi:
  • MapStore: Modern WebMapping con OpenLayers, Leaflet e React con Mauro Bartolomeoli
  • GeoServer, il server open source per la gestione interoperabile dei dati geospaziali con Andrea Aime
  • INSPIRE services con GeoServer ed HALE, state of the art con Simone Giannecchini

Inoltre, Mercoledì 28 Novembre dalle ore 11:30 alle ore 13:30 Simone Giannecchini terrà un workshop dal titolo "Infrastrutture per dati territoriali con i prodotti Open Source di GeoSolutions: introduzione e casi di successo" dove dopo una prima sessione di approfondimento sulla offerta di GeoSolutions  e sui propri prodotti Open Source verranno mostrati alcuni casi di successo a livello italiano ed internazionali di infrastrutture di dati geosapaziali create in collaborazione con GeoSolutions. Di seguito l'agenda dettagliata.

  • GeoSolutions, chi siamo e cosa facciamo
  • Introduzione ai prodotti Open Source di GeoSolutions: GeoServer, MapStore, GeoNode
  • Il SIT del Comune di Bolzano
  • La SDI del Comune di Genova
  • Un esempio di piattaforma DAAS (data-as-a-service) commerciale per il mondo Oil&Gas
  • Un esempio di piattaforma DAAS (data-as-a-service) commerciale per il mondo Earth Observation
  • La piattaforma IHP-WINS dell'UNESCO
If you are interested in learning about how we can help you achieving your goals with our open source products GeoServerGeoNodeMapStore and GeoNetwork as well as throw our Enterprise Support Services and GeoServer Deployment Warranty offerings, feel free to contact us!   The GeoSolutions team,
Categories: OSGeo Planet

gvSIG Team: gvSIG en Seguridad, Emergencias y Protección civil

OSGeo Planet - Wed, 2018-11-07 17:28

Cuando se trabaja con vidas humanas la elección de la tecnología no es algo baladí. Cuando se trata de analizar datos que pueden darnos la información necesaria para evaluar la seguridad ciudadana tampoco.

En las pasadas 14as Jornadas Internacionales de gvSIG hubo un par de sesiones temáticas dedicadas a la Seguridad, Emergencias y Protección Civil. Ponencias cuya grabación ahora ya tenéis disponible para su consulta.

Aplicar nuestros conocimientos y ver cómo se aplica la tecnología que impulsamos para proyectos como los que os traemos en este post nos produce a todo el equipo de la Asociación gvSIG una enorme satisfacción. El software libre, el conocimiento, al servicio de la gente, en las situaciones más críticas.

En primer lugar os enlazamos el vídeo a la ponencia titulada ‘Base de datos geográfica de artefactos explosivos improvisados’. Proyecto relacionado con la lucha antiterrorista que hemos abordado para MINUSMA (Misión Multidimensional Integrada de Estabilización de las Naciones Unidas en Malí).

Antes de seguir, una ponencia invitada a este post…la presentación del proyecto SIGAPRED que fue realizada unos días antes en las 5as Jornadas gvSIG de Uruguay. El motivo, la relación directa con el post, en ella se habla de la IDE implantada en  el Observatorio de Estudios Sobre Convivencia y Seguridad Ciudadana de la Provincia de Córdoba (Argentina)

Pasamos a una ponencia con una componente tecnológica ‘especial’, en la que hemos integrado en gvSIG Online algoritmos que permiten determinar la cobertura de REMER, la Red de Radio Emergencia de la Dirección General de Protección Civil y Emergencias.

El año pasado presentaron la puesta en marcha de gvSIG Online, y este año el Consorcio Provincial de Bomberos de Valencia nos muestra como utilizan una Infraestructura de Datos Espaciales en software libre para la prevención y gestión de emergencias. Muy recomendable ver la evolución del proyecto y su expansión a todo tipo de ámbitos relacionados con la actividad del CPB Valencia.

Geoscan, expertos en geología y SIG, nos mostraron como gvSIG ha sido utilizado como herramienta de apoyo en el estudio de deslizamientos

Llevamos varios años impartiendo cursos para la Escuela Nacional de Protección Civil de España. Es especialmente satisfactorio ver como los alumnos aplican gvSIG a temas como los accidentes de mercancías peligrosas en carretera

Seguimos con la presentación del policía local Gilberto Díaz, que nos presentó la ponencia ‘Perspectivas de gvSIG en materia de siniestralidad vial’

Y para acabar os traemos dos ponencias relacionadas con trabajos de investigación en los que estamos trabajando junto a expertos en estadística de la Universitat Jaume I, por un lado nuevos desarrollos en gvSIG para criminología que pronto estarán disponibles:

…y por otro el uso de gvSIG para el análisis de datos geoestadísticos en el 112 (número único de asistencia a la ciudadanía ante cualquier tipo de emergencia).

Y si llegados a este punto, os preguntáis cómo aún no estáis trabajando con la Suite gvSIG y colaborando con la Asociación gvSIG, y queréis contactar con nosotros…os dejamos un email info@gvsig.com

Categories: OSGeo Planet

Fernando Quadro: Nova Iorque: Dados abertos e Cidade Inteligente

OSGeo Planet - Wed, 2018-11-07 13:08

Prezados leitores,

Hoje vamos falar sobre dois assuntos que estão bastante na “moda”, que é Open Data (Dados abertos) e Smart Cities (Cidades Inteligentes). Tenho lido bastante sobre esses assuntos nos últimos tempos e tenho visto algumas coisas que sinceramente não tem me agradado muito aqui no Brasil.

O primeiro ponto, é que tenho visto algumas cidades (prefeituras) implantarem acesso Wi-fi gratuito a população e após isso se auto intitularem Smart City, o que eu discordo totalmente. O conceito de Smart City é muito mais amplo que apenas disponibilizar Wi-fi a população.

Eu acredito que não tem como se tornar uma Smart City sem a implantação de Open Data. E vou além, sem que a população seja envolvida de uma forma que a própria veja a importância e o retorno que esses dados abertos trazem a ela própria.

Um modelo que vejo como referência tanto em Smart City como em Open Data é Nova Iorque. Lá, tudo começou a muitos anos atrás:

Como podem ver na imagem acima Nova Iorque já tem a cultura de disponibilizar dados desde 1974, porém em 2012 foi criada uma lei que diz que em Nova Iorque é obrigatório a disponibilização de dados abertos! Aqui no Brasil, eu não vejo outro caminho para que isso possa se tornar uma realidade. Eu sei que já temos o decreto da INDE, que é um grande avanço para disponibilização de dados, principalmente quando falamos em dados geoespaciais, porém, eu gostaria de ver algo mais enérgico, com prazos determinados para implantação, mesmo que esses prazos para as agencias governamentais sejam mais amplos (10 anos por exemplo, na esfera Municipal).

Outro ponto que vejo em Nova Iorque, é que eles tem um departamento full time trabalhando para que a implementação e a implantação dos dados abertos funcionem, e acho que precisamos disso também aqui no Brasil, talvez algo ligado ao IBGE, mas que pense 24h em dados abertos, e que possa dar suporte a todas as instâncias do governo (Federal, Estadual e Municipal).

E por último e não menos importante, para dar certo é essencial que haja engajamento com a população, ou seja, inserir a população nesse cenário, criando aplicativos para que o próprio cidadão possa inserir informação, indicar quando uma informação estiver errada, e principalmente usufruir do resultados dessas informações processadas através de aplicativos que serão desenvolvidos e que irão melhor a vida do cidadão.

Gostaria deixar claro que tudo que escrevi acima, é a minha opinião, e se você tiver uma visão diferente, fique a vontade de deixar seu comentário para que possamos conversar e trocar uma ideia.

Vou deixar alguns links sobre a questão do Open Data em Nova Iorque, para aqueles que tenham interesse no assunto:

GovTech – How New York City Tells the Story of its Open Data Work

Harvard Datasmart – New York City Open Data a Brief History

Amny – Open data is changing how New York City Government Work

Categories: OSGeo Planet

gvSIG Team: 5as Jornadas gvSIG Uruguay. Ponencias, talleres y vídeos disponibles

OSGeo Planet - Wed, 2018-11-07 12:40

Ya se encuentran disponibles las presentaciones realizadas en las 5as Jornadas gvSIG Uruguay, que tuvieron lugar los días 18 y 19 de octubre en Montevideo.

También está disponible la grabación de las ponencias y de los talleres que se impartieron durante las jornadas para ser visualizados online.

Los talleres que se impartieron durante las jornadas fueron los siguientes: “R espacial y R con gvSIG”, “Scripting: Exprimiendo / Extendiendo gvSIG ” y “gvSIG Batoví”.

En el apartado de comunicaciones se puede encontrar el material necesario para poder seguirlos.

Si no pudiste asistir a las jornadas ya tienes todas las presentaciones y su grabación, así como una gran cantidad de material didáctico nuevo sobre gvSIG gracias a la grabación de los talleres.

Categories: OSGeo Planet

CARTO Inside Blog: An update on MVT encoders

OSGeo Planet - Wed, 2018-11-07 08:00

It has rained a lot since we last wrote about MVT encoders performance. Taking advantage of the recent Postgis 2.5 release, we decided to update our measurements and explain why we made the switch and started generating vector tiles in the database.

Why?

A few weeks ago we decided to change our main tiler (Windshaft) to use Postgis’ St_AsMVT instead of node-mapnik as the default technology to generate MVT. Some of the reasons behind this decision were:

  • Database servers are scaled depending on customer needs, that means that by encoding MVTs in those servers we have more resources available to generate tiles faster. This also minimizes the friction between different users sharing resources in the tiler.
  • Reduce the network load between the database and the tiler as small geometries are discarded early in the process.
  • Mapnik bindings are limited (node, c++, python) while the only requirement to use Postgis is a database connection. As new projects arise inside CARTO, we can share the same encoder independently of their programming language. In a similar way, once we update Postgis all those projects benefit automatically from it.
  • It is easier for CARTO to improve Postgis as we have contributed to the project more frequently than to Mapnik.
  • Better overall performance.
Animated Taxi Pickups map


How?

Our main concern when preparing the transition was to have the process be as seamless as possible, which meant having the tiles generated by St_AsMVT match the ones from Mapnik as close as reasonably possible.

Aside from multiple bugs fixed in Postgis 2.4.6 and 2.5, we introduced various performance improvements as a PARALLEL implementation. Tile by tile comparison even led us to find and address issues in our Mapnik stack.

Nevertheless, there are still some differences between the two renderers:

  • When working in PARALLEL, St_AsMVT duplicates keys and might introduce repeated values. This is discouraged in the spec, but valid nevertheless.
  • They have different behaviour when working with invalid geometries. Although it varies case by case, Mapnik tends to drop the invalid geometries and Postgis tries to fix them.
  • Depending on the encoding and simplification, Mapnik might not simplify the geometries as much as Postgis when retrieving them from the database so you might want to do an extra simplification step by passing the simplify_distance distance parameter to mapnik-vector-tile.
  • Mapnik assigns an id to each feature by generating it sequentially by tile. Postgis doesn’t add it as it’s optional per the MVT spec. In the next major release of Postgis (3.0) you will be able to assign it using a table column.
  • Mapnik orders feature properties alphabetically, while Postgis outputs them by their other in the database.
Performance comparison

As we mentioned before, one of the reasons that move us to use Postgis to generate vector tiles was performance and the graph at the header of the post shows the impact that it had in some of our platform benchmarks in production.

We set up a simple framework to test performance, not only to compare Mapnik and Postgis encoders but also different iterations of St_AsMVT.

Methodology Datasets

To be able to compare distinct geometry types, we loaded several public datasets into the database using CARTO’s Import API:

We used simple SELECT queries as any complexity added there would affect both encoders in the same way.

Requests

We launched http requests against the tiler, running each different one at least 50 times and 30 seconds, and discarded the first 5 iterations to reduce the impact of hard drive cache misses.

The requests cover a different amount of geometries per tile and, in some cases, different amount of columns to be able to see the impact of attribute encoding. Since we observed a clear difference when geometries are discarded because of their size, we also had different zoom levels.

Server

The specs of the machine are:

  • Intel i7-5820K with 6 cores and 12 threads. Postgres had 10 worker processes available.
  • 32GB of RAM. Postgres had 4GB available for shared buffers, enough to fit these tables.
  • Linux running PostgreSQL 11.0, Postgis 3.0.0dev r16981, GEOS 3.7.0, protobuf 1.3.1, node 9.11.1 and node-mapnik v3.6.2-carto.11.
  • Any setup required (Windshaft, PostgreSQL) was identical between runs except for the setting to switch between encoders.
Results

Here are some of the most representative graphs derived from our tests:

  • Point geometries had a similar performance:
Point requests


  • When we increased the number of columns per point we observed that properties encoding is faster (~1.2x) in Postgis:
Point requests


  • Line geometries were encoded faster in St_AsMVT: 0.5x to 1.5x improvement depending on the zoom level.
Point requests


  • Polygon geometries were faster in Mapnik (1.5x faster in some cases). When we analyzed it we observed that over 90% of the CPU cycles in St_AsMVTGeom went spent in St_MakeValid:
Point requests


  • Postgis was faster discarding small polygons. In the extreme cases where everything is discarded it, e.g. city boundaries in the tile [0/0/0], it can be up to 20x faster.
Point requests


Takeaways

One of the important aspects to consider when generating vector tiles is that their size and the time it takes to generate them it is highly tied to the number of properties that each geometry has associated. As seen above, the same point tile generated using St_AsMVT takes 9x more time when moving from 5 properties to 42. Consider optimizing your requests by simply removing unnecessary columns from the SQL queries.

A second element to consider is that since points aren’t simplified automatically by zoom level, the lower the zoom the more points were included in each tile. Point aggregations are a good way to work around this and improve performance by encoding too many points that are going to be rendered in the same place anyway.

Another idea we considered in the past was disabling polygon validation but we have discarded it as one single invalid polygon can pollute whole visualizations. It would be interesting to analyze why Postgis validation (based on GEOS) is way slower than Mapnik’s (based on Boost), and addressing this would benefit multiple SQL functions that use it, like St_IsValid.

As I final note, beware that Mapbox Vector Tile Specification version 3 is currently under development. It will bring new feature types and improvements to reduce tile size, but it will require adapting encoders and decoders which could impact performance.

Categories: OSGeo Planet

Marco Bernasocchi: The new QML widgets in QGIS – When widgets get unbridled

OSGeo Planet - Tue, 2018-11-06 06:30

Individuality is the definition of freedom. And freedom is the fundamental requirement of man’s mind. QGIS possibly cannot give you all the freedom you require in life. But at least a lot of freedom in how you manage your work. QGIS 3.4.0 LTR was released last week and it comes loaded with features supporting big freedom in the configuration of your projects.  Let’s focus on the QML Widget. QML is the smart casual look of widgets. With the help of some simple code, you will be able to visualize your data in the attribute form like never before. You can display beautiful charts, complex JSON data, and fancy colored unicorns. 

How it’s done

Let’s start with an example. In the Attribute Form configuration in the Layer Properties you have first to activate the drag and drop designer. Not only can you drag the field- and relation-items from the available widget list to the form layout list, but also a QML Widget. When you drop this item, it creates an “instance” of a QML Widget. This means, you can drag and drop as many QML Widgets as you like to have on your form and configure each of them individually.

QML (Qt Modelling Language) is a user interface specification and programming language. It allows developers and designers alike to create highly performant, fluidly animated and visually appealing applications. QML offers a highly readable, declarative, JSON-like syntax with support for imperative JavaScript expressions combined with dynamic property bindings. Source: Qt documentation

It’s a bit like an HTML page on the attribute form but very well integrated with Qt.
On dropping the item, the configuration dialog pops up. After closing you can come back to it by double-clicking the item in the form layout list, like you do it with containers and tabs. If your don’t know QML that much yet, the default snippets from the drop-down can create an example of a rectangle, a pie chart or a bar chart. This could help you to create your own widget. On the right you can see a preview of your widget in real time. There are powerful layout possibilities to design it according to your ideas. For more information about it see the QML layout part of the Qt documentation.

But a chart makes no sense if there is no data. Of course, you can enter the data directly into your QML code, but most likely you need the data of the features to be visualized. This brings us to expressions. You can use them like you are used to in Default Values, Constraints and Display Messages. You’ll find the well-known expression builder widget in this configuration as well.

Using expressions

So let’s assume we want to visualize who holds what share of a forest. These forests are owned by the country (national), the canton (cantonal) or private. To keep it simple we have three attributes for that: national_share,cantonal_share and private_share.
After creating the default pie chart you will find this snippet in the QML code text area:

PieSeries { id: pieSeries PieSlice { label: "First slice"; value: 25 } PieSlice { label: "Second slice"; value: 45 } PieSlice { label: "Third slice"; value: 30 } }

Let’s set the field expressions into the PieSlice-values. Just select them in the expression widget and add them with the + into the chart.

As you can see the expressions in the code are wrapped inside expression.evaluate("<expression>"). This means there are no limits in using expressions.
You are open to use more complex expressions like e.g. for the title property of the pie chart:

title: expression.evaluate("CASE WHEN @layer_name LIKE \"forest\" THEN \"Forest\" ELSE @layer_name END")

Or in case the task with forest shares would be solved with relations to other layers by filling up a model with the children and the children’s share. This is possible by using expressions with the help of the expression functionrelation_aggregate.
More information about expressions can be found in the QGIS Documentation.
Back to our example. The result will look like this on the attribute form. It visualizes the share values in the pie chart.

The visualization is not (yet) updated in real time when the values change. But this would be a nice thing to have in the future… If you would like to support this, please contact us.

And that’s it

I hope you liked reading and you will enjoy using it to make beautiful widgets and forms. If you have questions or inputs, feel free to add a comment.
… and in case you still asking where the promised unicorns are. Well you have to wait for the part 2 of this article

Categories: OSGeo Planet

QGIS Blog: Report back on the 20th QGIS Hackfest in Zanzibar

OSGeo Planet - Mon, 2018-11-05 20:35

 

 

This post summarizes personal reports of community members who attended the 20th Hackfest in Stone Town, Zanzibar the week before the FOSS4G in Dar Es Salaam.

Report from Matteo Ghetta

QGIS developers spent 3 days in the beautiful island of Zanzibar were they worked on bug fixing, improvement of new features and documentation enhancements. It is extremely important for developers to meet together and spend some days working side by side given that the geographical distribution is very wide.

Thanks to the (local government & Yves Barthelemy) we had the chance to visit the Land Mapping Commission in Stone Town: It has been an amazing experience for both developers and local council. Moreover developers visited the SUZA (University of Zanzibar) and saw the local infrastructure and GIS related workflow.

  • I have merged many PR of the documentation
  • Many old and not old issues have been closed or the author has been pinged
  • Many new additions to the documentation both small corrections and new sections from scratch
  • A new version of DataPlotly released
  • A small presentation of DataPlotly to the QGIS crew
Report from Mario Baranzini

I spent one day of the Hackfest contributing to the SLYR project, improving the conversion of LAB colors. SLYR (https://github.com/nyalldawson/slyr) is a project started and almost entirely developed by Nyall (which was not present to the Hackfest) which aims to provide tools to extract and convert symbols from ESRI .lyr and .style files and use them in QGIS. Nyall has recently worked hard on the project and perhaps soon sponsorship will allow further development.

Around this theme there is a keen interest. Even during the FOSS4G there has been discussion of the conversion of ESRI symbols and there was also a presentation of how the conversion in Israel is currently managed.

During the rest of the hackfest, I worked on the implementation of the new QField file selector.

Report from Denis Rouzaud Report from Tim Sutton

Firstly let me thank the QGIS project for contributing to my travel and accommodation costs – I am most grateful for the support! This hackfest was special for me because it is the first hackfest in an African country. As an African born person living in the interface between “first world” and “third world”, I have always had a particular social agenda with QGIS: To bring spatial tools to support responsible and sustainable management of our world. There is a huge technological and skills divide between Europe and less developed societies where QGIS can be a valuable social enabler in helping to advance the standard of living and quality of life. Hosting a hackfest in an environment where we don’t have tree cadastres, street furniture cadastres and every aspect of civilian life mapped and systematised is an important way to build empathy and understanding in our QGIS core community members for a broader cross-section of our user base. Having the opportunity to meet with users from ZMI (the Zanzibar Mapping Initiative) and students from the local university was a really uplifting experience. ZMI are building the national cadastre from the ground up using QGIS, PostGIS and UAV mapping. We were privileged to have a number of Zanzibar residents join us on the hackfest and get to experience just how appreciative they are, first hand.

There were a number of interesting topics that arose during the hackfest which I will try to summarise here:

Certification:

We held an extended meeting on the QGIS Certification Programme, mainly detailing how we should go about the review process for onboarding new organisations.

Mac OS Build:

I spent quite a bit of time struggling with my MacOS build on QGIS. It’s definitely an area of the project that needs more work as the process can be non-trivial and the brew based formulas quickly become outdated.

QField Show and Tell:

Matthias showed off the latest version of QField and all the hard work they have been putting into the QGIS mobile client. This Android based version of QGIS for mobile data gathering is a really great project and is getting more and more useful with each release. There was also a translation sprint for QField during which I translated it to Afrikaans.

QField Autobuilds:

Denis and Matthias showed off the work they have done to automate the .apk builds for QField. Their system combines Travis and some git hooks to automatically build .APK’s whenever a pull request os made or a tag is made. See the QField travis for details: https://github.com/opengisch/QField/blob/master/.travis.yml

Plotly plugin:

Matteo Ghetta showed off the latest capabilities of the Data Plotly plugin for QGIS 3. The plugin supports the creation of a wide assortment of charts from your layer’s attribute data. See the plugin homepage here for examples and more detail: https://github.com/ghtmtt/DataPlotly 

Governance:

I took the opportunity to sign off the statute changes from 2017 general meeting. This was my last official act as outgoing project Chair. I was also extremely humbled to be awarded an Honorary PSC membership during the FOSS4G2018 conference.

TimRecievesHonoraryPSCMembership
Easter Eggs:

I contributed a couple of new easter eggs to QGIS. While easter eggs are fun, the data behind these hidden tools provide an important history of the project. I took the opportunity of having many long-time QGIS community members around the dinner table to collate all the previous QGIS meet-up dates. You can view this as a map here: https://github.com/qgis/QGIS/blob/master/resources/data/qgis-hackfests.json 

The spatial clustering of these events shows that going to Zanzibar and getting out of our geographical comfort zone is a really useful endeavour.

Report from Paolo Cavallini

During the HF in Zanzibar I mainly worked on:

  • Plugins: I cleaned up the queue of unapproved plugins, contacting all individual authors, fixing what was possible and deprecating the worst cases; we also solved a long standing issue with a contentious plugin
  • Issues: I examined many tickets, especially those waiting for feedback
  • Training certification: through a very productive meeting we defined clear rules for accepting certifying organizations, with the main aim of driving people to actively support the project:
    • in the application process, the proponent should explain what are his contribution to QGIS project
    • following an initial review, the application will be sent to local QGIS groups for their opinion, which should take place in less than one monthly; where there is no user group, the responsibility will fall entirely on PSC shoulders
    • the training material for each course should be released with a free license, and a review will be done; if the material is not of adequate quality, this is a cause for refusal
    • then PSC will take a decision and publish the contributions as stated by the proponent, for transparency
  • Other meetings helped focusing on our mission, defining the relationship between volunteer and paid work, and other
  • I visited the Land Mapping Commission and the University, soliciting a tighter cooperation with QGIS project.
Report from Admire Nyakudya

During the heck fest in Zanzibar, I mainly worked on porting some common plugin we use to QGIS 3. The plugins I mainly worked on were Cogo Parcel plugin and Sg Diagram downloader. We visited the Zanzibar mapping agency and interacted with the people who were working on capturing cadastral data using QGIS and explained them about the COGO Parcel plugin which streamlines capturing cadastral boundaries in QGIS and storing the results in a PostgreSQL database. To summarise the work I was working on during the heck fest.

  • Port Cogo Plugin from QGIS 2 to make it compatible with QGIS 3. This has been achieved and now awaiting one of my team members to approve the pull request.
  • Expand functionality being offered by the COGO Parcel plugin to include the new reporting framework in QGIS 3.
  • Investigate ways to generate projects on the fly after talking to Matteo and seeing the work he has done with QGIS project generator.

I also went to the Zanzibar university to see ways in which QGIS is being used in the local university and how students are integrating drone imagery with QGIS.

Report from Matthias Kuhn

The highest value of hackfests is actually to meet and greet and discuss ideas and bigger plans. There is a lot of communication on a high bandwidth channel (also known as face-to-face) that strengthens the community. This helps to quickly overcome technical problems sometimes. Or to give someone some tips and tricks you normally wouldn’t come to (e.g. optimizing a git workflow) that results in a long-term gain because of sustainable productivity improvements. I tend to walk around once in a while and just randomly bump into people where I – more or less successfully – try to help them solve their issues or discuss approaches. In the same area, there is the possibility to informally discuss plans about organization, workflows and strategies. The concept of an LTR was to a big degree discussed and built at a hackfest in Essen some years ago. This year there was a very interesting discussion about the topic LTR. For how much time the “long” in the term “long term release” should stand, about the life cycle of releases and of long-term releases. One of the important things when it comes to this question is what organisations think and do – because, in the end, it is mostly for organisations that the LTR exists – and how development resources can be assigned to the task of keeping a high quality of an LTR during its whole lifetime. There are no conclusions yet, but synchronizing on these ideas is often the seed for tomorrow’s exciting changes.

We were also visited by a group of students from the local university. It was really refreshing to talk to them about open source and QGIS and how we work and what the challenges of spatial data and GIS infrastructure are on an island like Zanzibar.

Feature-wise, the main thing I worked on feature wise was a new snap to grid functionality that is available for digitizing tools. It allows configuring a precision for vector layers. Whenever a new node is added or an existing one is edited on this vector layer, it will be automatically placed on this grid. Normally this is used to force the objects on a layer to something like cm or mm resolution.

And then, of course, I did my daily bunch of (mostly volunteer) pull request reviewing, some code cleanup and some bugfixes.

Report from Marco Bernasocchi

https://photos.app.goo.gl/seX2kSJ9vKxKBZt58

The Zanzibar hackfest was a special one for me, my first hackfest as QGIS.org co-chair. My main goal was definitely to get as much information out of Tim as possible so that I wouldn’t have to keep on bugging him all time

Categories: OSGeo Planet

gvSIG Team: First gvSIG 2.5 build for testing is already available

OSGeo Planet - Mon, 2018-11-05 13:12

The first ‘build’ of the next gvSIG Desktop version for testing is already available. You can download it from the ‘Development versions downloads‘ section of the gvSIG Association website.

With this we encourage you to help us to stabilize the next  gvSIG Desktop 2.5. Mainly we ask you for testing these ‘builds’ during this error fixing phase and you tell us all the incidents that you find at the mailing list.

This first ‘build’ is available with installable distributions for Windows and Linux, and from the first RC (Release Candidate) there will be portable versions, including a distribution for Mac.

gvSIG Desktop 2.5 has a lot of novelties. Not all of them are included at this build (for that it’s not a RC yet). Among the novelties of this new version there are some of them that are remarkable, for example the report designer, the option to work with virtual fields, field importer, quick map exporter/generator, ring map,…

Besides it will integrate all the novelties that were published after 2.4 version. (black icons, option to select and count duplicate values, SIGPAC XML files loading, 3D vector layer into 2D layer conversor, plugin to create forms for gvSIG Mobile, plugin to capture coordinates, integration with gvSIG Mobile, Epanet, The Horton MachineStatistics Viewer, v.4 support of Cadastre GML INSPIRE in Spain and update of several plugins)

In the next weeks we will publish detailed information about the novelties of this version.

We look forward to your collaboration! Time to test!

Categories: OSGeo Planet

gvSIG Team: Primer build de gvSIG 2.5 para testeo ya disponible

OSGeo Planet - Mon, 2018-11-05 13:12

Ya tenemos disponible el primer ‘build’ para testeo de la próxima versión de gvSIG Desktop. Lo podéis descargar desde el apartado de versiones en desarrollo de gvSIG Desktop de la web de la Asociación gvSIG.

Con ello os animamos a que nos ayudéis a estabilizar lo que será el próximo gvSIG Desktop 2.5. Principalmente os pedimos que durante esta fase de corrección de errores probéis estos ‘builds’ y nos vayáis comunicando en la lista de usuarios todas aquellas incidencias que encontréis.

Este primer ‘build’ está disponible con instalables para Windows y Linux, y a partir de la primera RC (Release Candidate – Candidata a final) ya habrá versiones portables, incluyendo una distribución para Mac.

gvSIG Desktop 2.5 viene cargado de novedades. No todas están disponibles en este build (por eso todavía no es una RC). Entre las novedades de esta nueva versión se encuentran algunas tan llamativas como el diseñador de informes, el trabajo con campos virtuales, un importador de campos, exportador/generador rápido de mapas, ring map,…

Además integrará todas las novedades publicadas posteriormente a la versión 2.4. (iconos Black, selección y conteo de duplicados, carga ficheros XML SIGPAC, 3D a 2D, crear formularios para gvSIG Mobile, capturador de coordenadas, integración con gvSIG Mobile, Epanet, The Horton Machine, Visor estadístico, soporte v.4 del GML INSPIRE de Catastro y actualización de varios plugins)

En las próximas semanas iremos publicando información detallada sobre las novedades de esta versión.

¡Esperamos vuestra colaboración! ¡A testear!

Categories: OSGeo Planet

PostGIS Development: Howard Hughes Medical Institute

OSGeo Planet - Sat, 2018-11-03 00:00

As a software engineer at the Howard Hughes Medical Institute, I work on a collaborative neuron reconstruction and analysis software called CATMAID 1 (screenshot: 3), which is used for neuroscience research. We use PostGIS to represent neurons in a 3D space.

They consist of 3D points that reference their parent nodes or are the root [=soma of neuron] if they have no parent). Together with synapses, point clouds and TIN meshes for modeling compartments in a dataset, they model the spatial aspects of our neuroscience world. Users create those neuron reconstructions manually in a collaborative fashion plus segmentation programs can be used as additional data source. Using its spatial indices, PostGIS helps us to quickly query neurons in a particular field of view. The space of a single project contains sometimes 100s of millions of interconnected individual points. We also do bounding box intersection queries between neurons and compartment meshes, which then refine in the front-end by doing more precise intersection tests.

This software is used by quite a few research labs and as far as I know they all do their own hosting with a dedicated server and this is what we do as well. The reason being mainly that wth larger datasets, we benefit from machines with a lot of RAM (>256G), fast SSD/NVMe drives and many CPUs as well as fast local data access for e.g. image data.

Thanks so much for making PostGIS work well in non-GIS contexts too—-it makes my life much easier!

Categories: OSGeo Planet

MapTiler: Maps in Augmented Reality with JavaScript

OSGeo Planet - Fri, 2018-11-02 11:00
Thumb

At the TimeMachine conference, the MapTiler Team introduced the very first application showing maps in augmented reality running from the browser without a need to install any mobile app. No need to have a flagship Apple device, it runs on any mobile phone. Try it now!

Three simple steps to get maps in AR on your phone

Step 1: Open on your phone or tablet: maptiler.com/ar

Step 2: Point your camera at the printed sheet or at the picture below

Step3: Watch the magic!

MapTiler AR marker

The application works entirely with your computer screen, but you will get even better results if you print the sheet (in colors or on a monochrome printer).

In both cases, you will get a map of Venice you can zoom in and work with as with any other map. The date on the top shows the depicted year as the city development progress during the last 1000 years. It is a prototype of the TimeMachine Atlas project, which has been just announced.

The technology behind this application

Our AR app is in the phase of a tech demo for now, but you can already start building your own with our maps from MapTiler Cloud service. It offers global map data in vector format and hosting for own geodata. They can be turned into tiles using MapTiler Desktop and then uploaded to MapTiler Cloud, styled and used in any application with open-source JavaScript libraries or mobile SDKs.

Our current AR app is built around AR.js library, which is an open-source project using WebGL and WebRTC technology. The same code will run in the future also on devices with native AR support (ARCore & ARKit), once they are exposed via WebVR because our code runs directly in a web browser.

We used modified Mapbox GL JS library for displaying the map. Let us know if you are our customer and want to utilize the code in your project.

Try the Augmented Reality on your device now

The AR application runs on every mobile device with a camera using just the web browsers. Try it yourself at maptiler.com/ar!

Categories: OSGeo Planet

PostGIS Development: Vanguard Appraisals

OSGeo Planet - Fri, 2018-11-02 00:00

Vanguard Appraisals is new to the GIS world. In fact, we aren’t really in the GIS world; we just kind of brush up against it. We do mass property appraisal for entire county and city jurisdictions, and we develop software to collect, price and maintain values. We also host assessment data online so that homeowners can search and find property information much simpler from the comfort of their own home. Our software and websites are used in 7 states (IA, IL, MN, MO, NE, ND, SD).

Continue Reading by clicking title hyperlink ..
Categories: OSGeo Planet

Markus Neteler: GRASS GIS 7.4.2 released

OSGeo Planet - Thu, 2018-11-01 18:15

We are pleased to announce the GRASS GIS 7.4.2 release

What’s new in a nutshell

After a bit more than four months of development the new update release GRASS GIS 7.4.2 is available. It provides more than 50 stability fixes and improvements compared to the previous stable version 7.4.1. An overview of the new features in the 7.4 release series is available at New Features in GRASS GIS 7.4.

Efforts have concentrated on making the user experience even better, providing many small, but useful additional functionalities to modules and further improving the graphical user interface. Segmentation now support extremely large raster maps. Dockerfile and Windows support received updates. Also the manual was improved. For a detailed overview, see the list of new features. As a stable release series, 7.4.x enjoys long-term support.

Binaries/Installer download:

Source code download:

More details:

See also our detailed announcement:

About GRASS GIS

The Geographic Resources Analysis Support System (https://grass.osgeo.org/), commonly referred to as GRASS GIS, is an Open Source Geographic Information System providing powerful raster, vector and geospatial processing capabilities in a single integrated software suite. GRASS GIS includes tools for spatial modeling, visualization of raster and vector data, management and analysis of geospatial data, and the processing of satellite and aerial imagery. It also provides the capability to produce sophisticated presentation graphics and hardcopy maps. GRASS GIS has been translated into about twenty languages and supports a huge array of data formats. It can be used either as a stand-alone application or as backend for other software packages such as QGIS and R geostatistics. It is distributed freely under the terms of the GNU General Public License (GPL). GRASS GIS is a founding member of the Open Source Geospatial Foundation (OSGeo).

The GRASS Development Team, October 2018

The post GRASS GIS 7.4.2 released appeared first on GFOSS Blog | GRASS GIS and OSGeo News.

Categories: OSGeo Planet

gvSIG Team: IX Jornadas Ibéricas de Infraestructura de Datos Espaciales. Ponencias disponibles

OSGeo Planet - Wed, 2018-10-31 19:05

Todos los interesados en las Infraestructuras de Datos Espaciales tienen ya disponibles las ponencias presentadas en las JIIDE 2018, entre las que hubo una importante presencia de la Asociación gvSIG. Además de un taller en el que mostramos la facilidad de trabajo con gvSIG Online para crear geoportales, se impartieron 3 presentaciones que ya tenéis disponibles para consulta.

Las ponencias presentadas son un claro ejemplo de la madurez y rápida implantación que está teniendo gvSIG Online, en ámbitos tan relevantes como la gestión municipal, las emergencias, la seguridad ciudadana o la agricultura. El software libre ha dejado de ser una alternativa para posicionarse como la tecnología prioritaria en el mundo de las IDE.

Os dejamos los enlaces a nuestras ponencias:

Para cualquier duda que tengáis…o interés en implantar las soluciones gvSIG para poner en marcha vuestras IDE, podéis contactarnos en info@gvsig.com

Categories: OSGeo Planet

Even Rouault: SRS barn raising: 5th report

OSGeo Planet - Wed, 2018-10-31 11:17
This is the fifth progress report of the GDAL SRS barn effort. A lot of activity in PROJ developments has been done this month again.

New conversion and transformation methods referenced in the EPSG database have been integrated:
  • Projection methods: Lambert Conic Conformal (2SP Michigan) and Laborde Oblique Mercator
  • "Change of Vertical Unit", "Vertical offset" and other fixes relative to vertical component handling
  • "Axis order reversal"
  • "Affine parametric transformation", and its implementation in PROJ computation core
  • "Molodensky-Badekas" transformation, and its implementation in PROJ computation core
Regarding database-related activities:
  • Make concatenate operation building from proj.db robust to inverse sub-operations (the EPSG database lists chained operations, but does not indicate if the forward or reverse path must be taken)
  • Reference transformation grids available in the proj-datumgrid-* packages, such as NAD83 -> NAD83(HPGN) grids, OSTN15_NTv2_OSGBtoETRS.gsb
  • Add a celestial_body table and celestial_body property to ellipsoid, as a provision to handle non-Earth bodies
  • Add a text_definition column to geodetic_crs and projected_crs to allow definition by PROJ string (or WKT)
  • Add  the possibility of defining in 'other_transformation' a transformation defined by a PROJ pipeline
  • The createOperations() method that returns transformations between two CRS is now able to find pivot CRS when no direct transformation path exists. One notable fact to underline is that the pivot is not necessarily WGS84, and several candidate pivots are explored. When transforming from CRS A to CRS B, the database will be searched for all CRS C for which there is a referenced transformation from/to both CRS A and CRS B (advanced users may also be able to restrict the candidate(s) pivot to use)
  • Import PROJ.4 definitions contained currently data/IGNF into the database in a relational form. A script to directly use the official registry as the source, instead of the PROJ.4 strings that derive from it, has also been developed but is not used yet.
  • Import of the CRS definitions and transformations between horizontal CRS from the CSV files of the published Esri Projection Engine Database Documentation. The projected CRS are imported with their ESRI WKT definition directly put in the database, so ongoing work is in progress to be able to ingest this WKT1 variant in the WKT2-based form used internally by PROJ.
  • Update the CRS and transformations to EPSG dataset v9.5.4
The addition of the DerivedEngineeringCRS, DerivedParametricCRS and DerivedTemporalCRS classes mark the completion of the implementation of the full ISO-19111:2018 CRS modelling

A rather tedious task has consisted in optimizing the code (mostly by avoiding a lot of unnecessary instanciation of temporary C++ objects, involving looking frequently at disassembled output to locate them) to make the generated binary lighter. This has enabled an optimized build to go down from 3.1 to 2.6 MB

The Doxygen-generated documentation of the C++ API has been integrated with the general PROJ documentation in ReST format, with the Breathe module.
An initial C API that makes part of the C++ functionality available to C users has also been added.

Finally, a RFC has been written to officially submit this work for PROJ PSC approval (doubling the size of a code base and changing its language requirements is not a mundane detail). This RFC has now been officially adopted, and will make it possible to ultimately merge this work into PROJ master. You may skim through it to look at a few examples of the use of the projinfo utility that demonstrates part of the new capabilities developed.

Categories: OSGeo Planet

Fernando Quadro: Charter Member da OSGeo

OSGeo Planet - Tue, 2018-10-30 16:58

Prezados leitores,

Eu gostaria de compartilhar com vocês que hoje saiu o resultado da Eleição de Membros da OSGeo, na qual fui um dos nomes indicados. Esta é a segunda indicação que recebo para participar da eleição, a primeira aconteceu em 2011.

Para que não conhece a Fundação Geoespacial de Código Aberto (Open Source Geospatial Foundation – OSGeo) é uma organização sem fins lucrativos cuja missão é promover a adoção global de tecnologia geoespacial aberta por ser uma base de software inclusiva dedicada a uma filosofia aberta e desenvolvimento participativo voltado para a comunidade.

Dito isso, quero comunicá-los que hoje me tornei um “OSGeo Charter Member“, e pra mim é uma honra fazer parte do quadro de membros dessa fundação. Gostaria de aproveitar pra agradecer ao Jody Garnett que foi quem lembrou do meu nome e me indicou para a eleição.

Categories: OSGeo Planet

MapTiler: TimeMachine Atlas - history of Europe on the map with a timeline

OSGeo Planet - Tue, 2018-10-30 11:00
Thumb

The TimeMachine project allows traveling through the time by mapping the last 2000 years of history. The MapTiler Team joins the effort and contributes its know-how in the field of geographical data visualization and building online maps. With a prototype called The Venice Time Machine Atlas for EPFL Lausanne, it shows how historical data and maps can be presented to the broad audience.

TimeMachine project

The TimeMachine project is an educative collaborative effort mapping last 2000 years of history. There are about two hundred institutions involved ranging from museums and archives to high-tech companies working in the field of artificial intelligence and digitization. The MapTiler Team is involved as an expert in the field of modern as well as historical maps.

TimeMachine Atlas prototype

Our prototype of the TimeMachine Atlas is a Map of Venice, which is changing throughout ages - showing the last 1000 year of the city’s development. The demo project is available on the timemachineatlas.eu address.

The geographical data were supplied by university employees and equipped with a time stamp determining building start date and its duration. Technologically, TimeMachine Atlas is a follow up of our previous Roman Empire vector map.

The map is armed with a time slider, which automatically plays the city’s evolution based on historical data or can bring you to your favorite era. Users can compare this map with a modern map, satellite view or two other scanned historical maps. Data are available also in 3D and with additional information about selected buildings like name, time duration and others.

Technical details, collaboration and future

The TimeMachine Atlas technology is powered by our own OpenMapTiles toolset, which is an open-source project for serving vector tiles. It is available at openmaptiles.org and the source code is published under the free BSD / CC-BY license on GitHub.

The suggested data model for future development is based on OpenStreetMap tagging convention with added start_date and end_date tags. Using open- and well-known data format will boost further collaboration on the project. Open data format allows usage of a set of existing tools: e.g., vector tiles can be generated from GIS data (drawn in QGIS, supplied in Shapefiles) and turned into vector tiles for example by the upcoming MapTiler Desktop v10.

The data has no vendor lock-in and can be ported to various environments from which they can be served. Currently, they are hosted on the MapTiler Cloud service, which offers worldwide maps which can be easily customized to fit the old maps’ feeling, hosting for additional geodata and global satellite layer.

The whole project is collaborative, with open-data and open-source community-driven development. We hope that the map will grow with the contributions from other project partners working on time machines in Amsterdam, Budapest, Paris and other cities and countries.

Experience the web application yourself!

The TimeMachine Atlas prototype for Venice is available online at timemachineatlas.eu.

Categories: OSGeo Planet

QGIS Blog: QGIS 3.4 Madeira is released!

OSGeo Planet - Sun, 2018-10-28 12:11

We are pleased to announce the release of QGIS 3.4 ‘Madeira’! Madeira was the location of our developer meeting in February 2018.

Windows installers and Ubuntu/debian binaries are already out, and all the packagers are actively preparing packages for the other operating systems. We’ll keep you updated when different packages and installers become available.

QGIS 3.4 comes with tons of new features, as you can see in our visual changelog.

QGIS 3.4 will become the first LTR of version 3. Based on our current plans, 3.4 will replace 2.18 as LTR in February 2019.

We would like to thank the developers, documenters, testers and all the many folks out there who volunteer their time and effort (or fund people to do so). From the QGIS community we hope you enjoy this release! If you wish to donate time, money or otherwise get involved in making QGIS more awesome, please wander along to qgis.org and lend a hand!

QGIS is supported by donors and sponsors. A current list of donors who have made financial contributions large and small to the project can be seen on our donors list. If you would like to become and official project sponsor, please visit our sponsorship page for details. Sponsoring QGIS helps us to fund our six monthly developer meetings, maintain project infrastructure and fund bug fixing efforts. A complete list of current sponsors is provided below – our very great thank you to all of our sponsors!

QGIS is Free software and you are under no obligation to pay anything to use it – in fact we want to encourage people far and wide to use it regardless of what your financial or social status is – we believe empowering people with spatial decision making tools will result in a better society for all of humanity.

Categories: OSGeo Planet

gvSIG Team: Apertura 14as Jornadas Internacionales de gvSIG

OSGeo Planet - Thu, 2018-10-25 18:03

Economía y Productividad. Seguro que alguno de vosotros, cuando vio el lema de estas 14as Jornadas Internacionales de gvSIG, pensó…¿y esto de la productividad qué tiene que ver con un proyecto de geomática con software libre? Incluso algún despistado podría pensar que esto son unas jornadas de economía. Y lo son, en cierto modo.

Hasta ahora nuestros lemas destacaban bien la parte más técnica: geolocalizando las TIC, conoce el territorio gestiona la realidad; bien la parte más social, más solidaria del proyecto. Compartiendo el conocimiento, conocer para transformar…es posible, es real. En las 8as Jornadas el lema ‘Tecnología, Solidaridad, Negocio’ ponía el foco en la solidaridad como eje central sobre el que debería girar la ciencia y la economía. Esto sigue siendo así.

Pero con el paso de los años nos hemos encontrado con dos sucesos que van de la mano. Primero, el avance imparable del software libre, de proyectos como gvSIG, que dejan de ser una alternativa a ocupar espacios antes impensables. Hablamos de los grandes proyectos, de trabajar con las grandes organizaciones.

En segundo lugar, la reacción a esto del viejo modelo de especulación con el conocimiento. La reacción del software cerrado o privativo ha pasado por intentar apropiarse del discurso del software libre. Si bien no puede de sus conceptos, lo intenta con las etiquetas. Si no puede con su ética, se apropia de la estética. Más de uno recordará el ‘…is open’, para presentar uno de los software que hace años monopolizaba el mercado.

Pero es que este año, sus conferencias (que casualidad, son estos mismos días), se anunciaban con la frase ‘Comparte tu trabajo. Inspira a la comunidad’.

La verdad es que no deja de resultar curioso. Las jornadas de la compañía de referencia de las soluciones cerradas hablando de compartir y comunidad, y una de las jornadas de soluciones de geomática libre de referencia a nivel internacional hablando de Economía y Productividad.

En un primer vistazo se podría pensar que cada ‘uno’ se quiere acercar a los valores del ‘otro’. Pero si algo nos enseña la ciencia, la Academia (donde estamos), es que debemos ser rigurosos, debemos ir más allá de las etiquetas con las que cada uno nos presentamos, porque eso son los lemas, etiquetas, puestas en escena. Debemos analizar con detalle.

Así que vayamos a las esencias. Aumentemos el nivel de abstracción. Cuando una organización, la que sea, basa gran parte de su modelo de negocio en la venta de licencias, de soluciones cerradas ¿que sentido tiene hablar de compartir? ¿Compartir productos capados que generan dependencia? ¿tecnologías limitadas por la imposibilidad del acceso al conocimiento?

¿Y qué sentido tiene de hablar de inspirar a la Comunidad? Con soluciones cerradas, las Comunidades de las que habla siempre serán comunidades limitadas. ¿Qué se quiere? ¿Inspirar comunidades limitadas?

Ahora situémonos en proyectos libres como los de la Asociación gvSIG. Y que recordemos una vez más si es necesario: Que ser libre no implica menor o mayor calidad de software sino acceso al conocimiento. Derecho de utilización, modificación y distribución.

Y preguntémonos, ¿Cómo se es más productivo? En soluciones que comparten y dan acceso al conocimiento, que se construyen conjuntamente, que permiten invertir sólo en aquello que necesitamos, que impulsan empresas locales expertas en tecnología, que permiten reutilizar, que nos hacen soberanos en la toma de decisiones, que reducen asimetrías o en productos que nos limitan, que especulan con el conocimiento, que generan gasto, que nos hacen dependientes de proveedores únicos que determinan nuestro futuro tecnológico, única y exclusivamente, en base a sus políticas comerciales.

Echemos un vistazo a las realidades que se van a presentar en estas jornadas, a los talleres, aplicaciones a sectores tan relevantes como la agricultura o el medio ambiente, tan críticos como la seguridad y la protección civil. Miremos los proyectos que se han presentado durante estos 14 años de jornadas internacionales que llevamos en Valencia (y en tantas otras, unas 50, en Francia, Alemania, Italia, Rusia, México, Argentina, Brasil, Uruguay, Venezuela, Paraguay, Perú y Chile ) y todo a pesar del estigma y las falsedades que se han querido hacer contra las soluciones libres, contra proyectos como el propio gvSIG. ¿Podemos hablar de economía y productividad?

No sólo podemos sino que debemos empezar a reivindicar el impacto que tiene la adopción de proyectos de software libre como gvSIG en la economía, en la productividad. Si el foco lo centramos ahí. Si conseguimos esto, entonces se podrán generar cambios de mayor calado, a nivel estructural.

Y aquí lo dejamos…

Bienvenidos a todas y todos a las 14as Jornadas Internacionales de gvSIG.

Categories: OSGeo Planet

Fernando Quadro: Junção espacial 5x mais rápida com PostGIS

OSGeo Planet - Thu, 2018-10-25 10:30

Hoje veremos como é possível aumentar a performance de sua consulta espacial com join, de uma maneira de certa forma “estranha”. Para este teste de desempenho com o PostGIS onde queremos saber quantos pontos estão dentro de cada polígono, temos: Uma coleção de polígonos de tamanhos variáveis ​​e uma coleção de pontos.

Esse teste é uma boa maneira de testar indexação, cálculos de pontos em polígonos e sobrecarga geral.

1. Configuração

Primeiro baixe alguns polígonos e alguns pontos:

Países
Lugares

Carregue os shapefiles em seu banco de dados:

shp2pgsql -s 4326 -D -I ne_10m_admin_0_countries.shp countries | psql performance shp2pgsql -s 4326 -D -I ne_10m_populated_places.shp places | psql performance

Agora estamos prontos com 255 países e 7343 lugares.

Uma coisa a notar sobre os países é que eles são objetos bastante grandes, com 149 deles tendo vértices suficientes para serem armazenados em tuplas TOAST.

SELECT count(*) FROM countries WHERE ST_NPoints(geom) > (8192 / 16);

2. Desempenho de baseline

Agora podemos executar o teste de desempenho de baseline.

SELECT count(*), c.name FROM countries c JOIN places p ON ST_Intersects(c.geom, p.geom) GROUP BY c.name;

Essa consulta levou 25 segundos. Se você colocar o processo em um profiler enquanto o executa, descobrirá que mais de 20 segundos são gastos na função pglz_decompress, e não fazendo algoritmos espaciais ou geometria computacional, apenas descomprimindo a geometria antes de entregá-la ao processamento real.

Porém, existem maneiras inteligentes de evitar essa sobrecarga:

  • Correção do PostgreSQL para permitir a descompactação parcial de geometrias.
  • No formato de serialização incluir uma chave hash exclusiva na frente das geometrias.

Essas são maneiras legais de manter a compactação para geometrias grandes e ser mais rápido ao alimentá-las.

No entanto, eles ignoram uma abordagem mais brutal e facilmente testável para evitar a descompressão: apenas não comprimem em primeiro lugar.

3. Um truque estranho

O PostGIS usa a opção de armazenamento “principal” para seu tipo de geometria. A opção principal tenta manter as geometrias na tabela original até que elas fiquem muito grandes, depois as compacta e as move para TOAST.

Existe outra opção “externa” que mantém as geometrias, e se elas ficarem muito grandes, são movidas para TOAST descomprimidas. O PostgreSQL permite que você altere o armazenamento em colunas em tempo de execução, portanto não é necessário hackear ou codificar para tentar isso.

-- Altere o tipo do storage ALTER TABLE countries ALTER COLUMN geom SET STORAGE EXTERNAL; -- Force a reescrita da coluna UPDATE countries SET geom = ST_SetSRID(geom, 4326); -- Execute novamente a query SELECT count(*), c.name FROM countries c JOIN places p ON ST_Intersects(c.geom, p.geom) GROUP BY c.name;

A junção espacial agora é executada em menos de 4 segundos .

Qual é a penalidade?

  • Com um armazenamento “principal” a tabela + toast + index é de 6MB.
  • Com um armazenamento “externo” a tabela + toast + index é de 9MB.

4. Conclusão

Para uma penalidade de armazenamento de 50%, em uma tabela que possui objetos muito maiores do que a maioria das tabelas espaciais, alcançamos uma melhoria de desempenho de 500%. Talvez não devêssemos aplicar compressão à nossa geometria?

Usar o armazenamento “principal” foi principalmente uma chamada de julgamento quando decidimos, não foi aferido nem nada – é possível que estivéssemos errados. Além disso, apenas objetos grandes são compactados; uma vez que a maioria das tabelas é cheia de pequenos objetos (linhas curtas, pontos), mudar para “externo” por padrão não teria qualquer efeito no tamanho do armazenamento.

Este post foi traduzido e adaptado livremente do post originalmente escrito por Paul Ramsey, do blog CleverElephant.

Fonte: Blog CleverElephant

Categories: OSGeo Planet
Syndicate content