OSGeo Planet

Fernando Quadro: Conheça o OpenStreetCam

OSGeo Planet - Mon, 2017-03-27 13:54

OpenStreetCam (originalmente chamado de OpenStreetView) é um projeto cujo objetivo é criar um grande banco de dados aberto com fotografias de ruas. Fotos georreferenciadas podem ser colocadas no sistema por qualquer um que abra uma conta de usuário.

Todos as fotos são supervisionadas por um certo número de pessoas (normalmente 3) para que se garanta que são adequadas ao projeto. Uma vez que uma foto tenha sido surpervisionada, ela torna-se disponível para as consultas da API e aparece no site para qualquer um vê-la, autenticado ou não. Em apenas 4 meses, a comunidade contribuiu com quase 12 milhões de imagens cobrindo 322 mil quilômetros.

O rápido crescimento do projeto atraiu a atenção do Google, que detém a marca registrada Street View. Eles demonstraram interesse ​​no OpenStreetView, mas também expressaram preocupações sobre o nome poder criar confusão, e depois de considerar os prós e contras com cuidado, foi decidido então mudar o nome do projeto para OpenStreetCam.

Se você ainda não experimentou o OpenStreetCam, por que não baixar os aplicativos gratuitos e abertos para Android ou iOS, e começar a contribuir com esse projeto?

Fonte: OSM Blog

Posts RelacionadosSovrn
Categories: OSGeo Planet

BostonGIS: PGConfUS 2017 Getting Stuff done in PostGIS

OSGeo Planet - Sun, 2017-03-26 04:42

A reminder, PGConfUS 2017 conference is just days away, and we'll be giving a training March 28th 2017, Jersey City, NJ at 1 PM. If you are coming, keep an eye on this page PGConf 2017 US Getting Stuff done with PostGIS materials.

If you haven't signed up already, there are still spots, make sure to buy your tickets at http://pgconf.us/conferences/2017#registration.


Continue reading "PGConfUS 2017 Getting Stuff done in PostGIS"
Categories: OSGeo Planet

GIScussions: Esri isn’t evil

OSGeo Planet - Fri, 2017-03-24 16:42

Yup you read it here, an open source advocate said that the leading GI proprietary vendor is not evil. Let me go one step further and say that I think Esri are pretty damn good. Now that is not to say that there are not aspects of their business model and practices that I wouldn’t criticise but more of that in another post.

Come to the Dark Side.

So what has prompted this outburst from an open source advocate? A mail from Suchith Anand to the OSGeo Discuss mailing list entitled “Is it possible for properitery GIS vendor to market thier properitery product as Open ?” (sic).

Hi all,

I have a query. If a properitery GIS vendor starts marketing thier properitery products as Open platform and software then what rights do the organisations and customers have who are mislead buying the  properitery software thinking it is open have ?  The definision of Proprietary software is very clearly defined, so  how can it be possible for any properitery GIS vendor to market their  software knowingly as open platform if it is properitery?

This also greatly affects the business and revenues of true open source software companies .  Who is responsible for any misleading marketing that results in losses to both customers who are mislead to buy the properitery software thinking it is open  and also to other companies who do true open source business who lose out on the business opportunities? Is it right business ethics to do this?

Best wishes,
Suchith

This has prompted several thoughts which I want to share in a wider format than the OSGeo mailing list (I will post this blog back to the list) partly because the list tends to be a bubble of OSGeo geeks and activists and I want to share these thoughts with less committed users or those who have not yet chosen open source.

I am going to try to deconstruct the propositions of the initial mail (others, particularly Jody Garnett have also expressed contrary views in the mail thread).

a properitery GIS vendor” I am pretty sure that this unnamed vendor is Esri inc and that the mail was prompted by Esri’s Open Vision which would understandably cause a few ruffled feathers and raised eyebrows amongst open source advocates.

The argument seems to be that proprietary software cannot be open, in my opinion that is patently incorrect. We use the word ‘open’ with regard to software and services in numerous different ways e.g. Open Source, Open Standards, Open Data, Open Access and some people even confuse or conflate open with free. The open source community does not have a copyright or any other ownership of the ‘open’ and cannot claim to be the arbiter of who can use the adjective ‘open’ or in what context. As a community we cannot even agree amongst ourselves on the differences between ‘open’ ‘free’ and ‘libre’ (watch this very good presentation on the Free, Open & Libre by María Arias de Reyna).

Furthermore the use of ‘open’ in Open Standards is specifically intended to encourage all (proprietary or open source) to support standards that enable data sharing, as the OGC’s welcome says:

The OGC (Open Geospatial Consortium) is an international not for profit organization committed to making quality open standards for the global geospatial community. These standards are made through a consensus process and are freely available for anyone to use to improve sharing of the world’s geospatial data.

Our members come from government, commercial organizations, NGOs, academic and research organizations.

Open Standards facilitate interoperability and limit ‘vendor lock-in’ which would hardly work without major proprietary vendors supporting the OGC and its standards. ‘Open’ is not the exclusive terminology of the open source community. Whilst we may not like it proprietary software can be described as ‘open’ even if the source code is not freely available.

And let’s be honest about this, apart from a very small number of users who really wants access to source code or has the ability to fix or enhance the code? For many users of open source software the cost of ownership is a major factor and in that case the ‘free’ bit is probably more important than the ‘open’ bit (although the cost of ownership is much more than the license cost).

If a properitery GIS vendor starts marketing thier properitery products as Open platform and software then what rights do the organisations and customers have who are mislead buying the  properitery software thinking it is open have?” Most businesses marketing software (regardless of whether it is open source or proprietary) consider that their potential clients have made a mistake if they eventually succumb to the sales patter of a competitor, we all think we have the ‘best’ offer (specification, quality, value). I doubt that many organisations that purchase Esri software consider themselves to have been mislead by their marketing materials about openness.

Sometimes the open source community (and I include myself in this mea culpa) can position themselves on the side of the angels in a battle between good and evil. That is bullshit! We deliver solutions to customers’ needs that hopefully enable them to harness the power of geography to manage assets, make better decisions and inform stakeholders, the choice between proprietary and open source software is made by informed buyers on the basis of what is best for their organisation in terms of functionality, service, lifetime cost and a whole range of other criteria. Sometimes proprietary solutions are the best answer and sometimes (increasingly so in my opinion) open source represents the most scalable and cost effective solution. The geo market is shifting rapidly from enterprise software implementations to services in which the underlying technology is less visible and less important to the buyer.

Rather than trying to wrestle over the usage of ‘open’ by Esri, open source businesses could learn a fair bit from the effectiveness of Esri’s marketing in building such a successful business that dominates our market. Along the way, whilst making profits (not a dirty word) Esri and in particular Jack Dangermond have done an enormous amount to promote the use of GI, provided employment for thousands of talented people and created the GIS category which has enabled many other business to prosper. They also give the OSGeo a nice big target to aim at!

 

 

Categories: OSGeo Planet

gvSIG Team: The 2nd gvSIG Festival is underway!

OSGeo Planet - Fri, 2017-03-24 09:48

After the successful first gvSIG Festival in the last year, the virtual gvSIG conference, a second edition will be held this year, in May 16th and 17th. The novelty of this year is that users can send their proposals about projects done with gvSIG, that will convert this initiative in a more global and open event.


This event is free of charge and completely online, through the webinar service of the gvSIG Association, with the advantage to count with speakers from different countries and where users and developers from any part of the world can hold them.

If you have done any project with gvSIG and you want to present it at the gvSIG Festival, you can send a summmary to the following e-mail address: conference-contact@gvsig.com, explaining the project. The summary will have no more than 300 words, it has to be written in Spanish or English, and you have to indicate the title and the language of the presentation.

Once the program is configured it will be published at the event website, and registrations will be opened.

We expect your proposals!


Filed under: community, english, events, training Tagged: gvSIG Festival
Categories: OSGeo Planet

gvSIG Team: ¡El 2º gvSIG Festival ya está en marcha!

OSGeo Planet - Fri, 2017-03-24 09:44

Tras el éxito del año pasado del primer gvSIG Festival, las jornadas virtuales sobre gvSIG, se va a celebrar este año una segunda edición, los próximos días 16 y 17 de mayo, en las que como novedad de este año los usuarios podrán enviar sus propuestas de proyectos sobre la aplicación, lo que convierte esta iniciativa en un evento todavía más global y abierto.


Este evento es gratuito y completamente online, a través del servicio de webinar en la Asociación gvSIG, con la ventaja de poder contar con ponentes de diferentes países, y donde las ponencias pueden ser seguidas por usuarios de cualquier parte del mundo.

Si has realizado algún proyecto con gvSIG y quieres presentarlo en el gvSIG Festival, puedes enviar un resumen explicando en qué consiste a la dirección de correo conference-contact@gvsig.com. El resumen deberá ser en español o inglés, de un máximo de 300 palabras, y en él se deberá indicar el título de la ponencia y el idioma en el que se daría.

Una vez configurado el programa se publicará en la web del evento y se abrirán las inscripciones para cada presentación.

¡Esperamos vuestras propuestas!


Filed under: community, events, spanish, training Tagged: gvSIG Festival
Categories: OSGeo Planet

Tom Kralidis: pygeometa: new release, hello YAML

OSGeo Planet - Thu, 2017-03-23 20:13
Metadata should be automagic and low barrier. pygeometa is a handy little metadata generator tool which is flexible, extensible, and composable.  Command line, or via the API, users can generate config files, or pass plain old Python dicts, ConfigParser objects, etc. We’ve just released 0.2.0 which supports WMO Core Metadata Profile output, as well as […]
Categories: OSGeo Planet

gvSIG Team: Aprende a trabajar con Modelos Digitales del Terreno y geoprocesamiento ráster con este vídeo-tutorial

OSGeo Planet - Thu, 2017-03-23 09:19

Complementando el post en el que os enseñamos los secretos del geoprocesamiento vectorial, hoy veremos como podemos aplicar algunos de los algoritmos disponibles en gvSIG Desktop a las capas ráster en general, y a los Modelos Digitales del Terreno en particular.

De los más de 350 geoprocesos disponibles en gvSIG, un buen número de ellos son aplicables a capas ráster, permitiéndonos hacer cálculos de todo tipo y que son especialmente útiles para algunas disciplinas científicas como la hidrología.

Mediante una serie de ejercicios prácticos, que podréis replicar en vuestra casa, este vídeo-tutorial os mostrará en pocos minutos como realizar geoprocesamiento ráster. Seguid leyendo….


Filed under: gvSIG Desktop, spanish Tagged: geoprocesamiento, hidrología, MDT, Modelo Digital del Terreno, raster
Categories: OSGeo Planet

GeoServer Team: GeoServer 2.11.0 Released

OSGeo Planet - Thu, 2017-03-23 08:37

We are happy to announce the release of GeoServer 2.11.0. Downloads are available (zipwardmg and exe) along with docs and extensions.

This is the latest stable release of GeoSever recommended for production system. This release is made in conjunction with GeoTools.

This release is made by Andrea Aime and Alessandro Parma from the GeoSolutions team. We would like to thank these volunteers  and everyone who contributed features, fixes and time during the release process.

Highlights of this release are featured below, for more information please see the release notes (2.11.02.11-RC1 | 2.11-beta ).

Change summary since RC1

If you already had a look at 2.11-RC1 announcement, here is an executive summary of changes since RC1:

  • Fixed inability to edit layer groups after modifying a group layer order and layer default style
  • Windows service installer now sets up enough memory for GeoServer to run (for production usage you might still want to tweak it)
  • Fixed interactin between layer group and security that sometimes prevented GetMap from succeding
  • Assorted JMS clustering fixes

See the full changelog for details. Continue reading for an extended presentation of all the new features in 2.11.0

Highlights

As this is a major release, GeoServer 2.11.0 contains a number of important changes, including both new features and compatibility requirements.

Improved loading and OGC request times for large installations

GeoServer has historically had issues with long startup times on installations having several thousands layers. A group of recent improvements significantly sped up these times making it possible to handle tends of thousands of layers with no particular headaches.

For reference, some tests were run on a Core i7 860, 16GB of memory, 2TB hybrid drive (spinning disk plus SSD cache), on two different data directories:

  • “Many states”: 1 workspace, 1 store, 10k layers, 10k cached layers
  • “Large”: 1001 workspaces, 11000 stores (a mix of shapefiles, postgis, directory of shapefile, single tiff, arcgrid, mosaics), 42000 layers and 42000 associated tile layers

Startup times have been measured in both “cold” and “warm” mode, “cold” means the operating system file system cache contains none of the configuration files and the startup has to actually read everything from the disk, “warm” means the data dir is fully cached in memory instead:

  • Many states, cold startup: 30s
  • Many states, warm startup: 21s
  • Large, cold startup: 107s
  • Large, warm startup: 45s

As you can see, worst case is loading in excess of 40 thousands layers in less than a minute and a half!

For reference, “Large” startup times with JDBCConfig are 290s for cold startup, and 120s for warm startup. This is mostly due to cached layers loading, if your configuration has none the startup time will be of around 20-30 seconds instead (no matter how many layers are configured, because JDBCConfig loads layers on a as-needed basis at runtime… for a cost, see below).

It is also interesting to compare the times needed to run a GetMap against one of the “topp:states” in the “many states” data directory.

Version Throughput req/sec Avg resp. time ms 2.10.1 169 47 2.10.1 + JDCBConfig 68 117 2.11-beta 233 34

More information can be found in the initial proposal, and on the latest mailing list thread about parallel catalog loading.

Thanks to Andrea Aime for this great result.

Improved lookup of EPSG codes

Ever been annoyed by some shapefile with a .prj declaration that GeoServer does not understand? If so, try this new release. We pulled some extra heuristics and smarts to figure out the equivalent EPSG code of thousands more cases. This will hopefully reduce guesswork when configuring new vector layers, not to mention significantly speeding up importing large directories of shapefiles in the “Importer” extension.

Selezione_198

Thanks to Andrea Aime for this much appreciated usability improvement.

Underlying labels in SLD

A new vendor option allow to underline labels in SLD (and CSS). Just add:

<VendorOption name="underlineText">true</VendorOption>

Selezione_197

Thanks to Nuno Oliveria and GeoSolutions for this new feature.

Opaque Container Layer Group Mode

A new layer group mode has been added, called “Opaque Container”. It’s similar to “single” in that you cannot see the layers contained in it, but it also prevents its layers from showing up at the root level of the capabilities document. This new mode is targeted to building “base maps” that the client is not allowed to take apart into their components.

You can read more about layer group modes in the GeoServer user guide.

Thanks to GeoSolutions and City of Helsinki for sponsoring this new feature.

Layer Group Security

The build-in data security can now secure layer groups too, for both global and workspace-specific groups. “Tree” mode groups (named tree, container tree, opaque container) will extend the security rules applied to them to the layers contained.

 

Selezione_196

Thanks to GeoSolutions and City of Helsinki for sponsoring this new feature.

REST API Security Improvement

New REST call allowing a users to modify his/her password (no UI for it yet, sorry). Here is an example cURL command:

curl -i -XPUT -H "Content-type: text/xml" -d '<passwd><newPassword>qqq</newPassword></passwd>' -u userName:currentPassword http://host:8080/geoserver/rest/security/self/passwordInviato il:09:48Da:Emanuele Tajariol

Thanks to Emanuele Tajariol and GeoSolutions for this work.

Extensions

GeoServer extensions are available as optional downloads adding capabilities to the core application.

YSLD module graduated to supported

The YSLD styling language has been graduated to supported land, becoming an official extension. YSLD is a YAML based language which closely matches the structure of SLD, and the internal data model that GeoServer’s renderer uses. Here is an example from the YSLD Cook Book:

title: 'YSLD Cook Book: Simple Line' feature-styles: - name: name rules: - symbolizers: - line: stroke-color: '#000000' stroke-width: 3

Compared to SLD it sports a similar structure, but in a more compact and readable syntax. High-level elements are optional, so the same example can be expressed in a more compact format:

line: stroke-color: '#000000' stroke-width: 3

Similarly to SLD and CSS, the user guide contains both a reference and a cookbook to get you started. There is also a styling workshop with both CSS and YSLD examples (including an important example for converting your SLD styles to YSLD).

Thanks to Boundless for contributing this module and pushing it to supported status.

Vector tiles module graduated to supported

The vector tiles community module has been graduated to extension status. The module allows generation of vector tiles out of WMS and WMTS requests, using SLD to filter the contents of the tile at the requested scale. Supported vector tiles formats are MVT (MapBox Vector Tile), GeoJSON vector tiles, and TopoJSON vector tiles.

This allow for much more compact data transfers, reduced tile caches thanks to overzooming support, and allows the client to control styling. Here is an example of the client rendering a world map as a vector tile, and highlighting a specific area by attribute:

vectortileoutputafrica

Hop on the documentation to get more information about this extension, along with a couple of presentations highlighting vector tile usage.

Thanks to Boundless for contributing this module and pushing it to supported status.

Upgrade Notes

This section highlights functionality that has changed from prior releases.

SLD 1.0 LabelPlacement fix default AnchorPoint

An issue with SLD 1.0 rendering has been fixed, this may affect the appearance of you WMS GetMap results.

This issue occurs in the following TextSymbolizer example, where the anchorPoint for the label placement has not been defined.

<sld:TextSymbolizer> <sld:Label><ogc:PropertyName>name</ogc:PropertyName></sld:Label> </sld:TextSymbolizer>

As shown below GeoServer 2.10 and earlier used the incorrect default anchor point. The default anchor point of of x=0.5 and y=0.5 generated a label at the geometry centroid which is more suitable to PolygonSymbolizers.

label_anchor_before

Prior TextSymbolizer anchor point default

For GeoServer 2.11 this mistake has been corrected, the default anchor point is x=0.0 and y=0.5 which is more suited to PointSymbolizers.

TextSymbolizer correct anchor point

Correct TextSymbolizer anchor point default

Alternative: Provide a Label Placement

To avoid making use of the default anchor point we recommend defining an label placement. For PolygonSymbolizers the following example is appropriate:

<sld:TextSymbolizer> <sld:Label><ogc:PropertyName>name</ogc:PropertyName></sld:Label> <sld:LabelPlacement> <sld:PointPlacement> <sld:AnchorPoint> <sld:AnchorPointX>0.5</sld:AnchorPointX> <sld:AnchorPointY>0.5</sld:AnchorPointY> </sld:AnchorPoint> </sld:PointPlacement> </sld:LabelPlacement> </sld:TextSymbolizer>

For PointSymbolizer we recommend both an anchor point and a displacement (to prevent overlap with the location graphic).

<sld:TextSymbolizer> <sld:Label><ogc:PropertyName>name</ogc:PropertyName></sld:Label> <sld:LabelPlacement> <sld:PointPlacement> <sld:AnchorPoint> <sld:AnchorPointX>0.0</sld:AnchorPointX> <sld:AnchorPointY>0.5</sld:AnchorPointY> </sld:AnchorPoint> <sld:Displacement> <sld:DisplacementX>8.0</sld:DisplacementX> <sld:DisplacementY>0.0</sld:DisplacementY> </sld:Displacement> </sld:PointPlacement> </sld:LabelPlacement> </sld:TextSymbolizer>

These changes explicitly centre polygon labels, and offset point labels (with an appropriate displacement allowing their graphic to be visible).

label_anchor_defined

TextSymbolizer with defined anchor point

Alternative: Restore the Previous Default

If you need to “restore” the previous incorrect default value, add the startup parameter:

-Dorg.geotools.renderer.style.legacyAnchorPoint=true

This setting is provided as an alternative to ease the upgrading process.

And more …

There are many improvements to look at in the release notes, cherry picking a few here:

  • Support JSON encoded exceptions for WFS 2.0 too (was already available for 1.0 and 1.1)
  • Drastically reduced output size for WMS PDF output format, when using a graphic fill based on repeated marks or SVG symbols
  • Improved scalability when using “advanced projection handling” thanks to a new implementation of SoftValueHashMap (used as a cache by the referencing subsystem)
  • Better transparency handling of NODATA and ROI (Region Of Interest)
  • Fixes in computation of global disk quota (if you find you have a negative number, upgrade, wipe out the quota database, and restart)
About GeoServer 2.11

Articles, docs, blog posts and presentations:

  • OAuth2 for GeoServer (GeoSolutions)
  • YSLD has graduated and is now available for download as a supported extension
  • Vector tiles has graduate and is now available for download as an extension
  • The rendering engine continues to improve with underlying labels now available as a vendor option
  • A new “opaque container” layer group mode can be used to publish a basemap while completely restricting access to the individual layers.
  • Layer group security restrictions are now available
  • Latest in performance optimizations in GeoServer (GeoSolutions)
  • Improved lookup of EPSG codes allows GeoServer to automatically match EPSG codes making shapefiles easier to import into a database (or publish individually).
Categories: OSGeo Planet

Jackie Ng: React-ing to the need for a modern MapGuide viewer (Part 15): Play with it on docker

OSGeo Planet - Wed, 2017-03-22 16:22
Today, I found a very interesting website from the tech grapevine:

http://play-with-docker.com

What is this site? It is an interactive docker playground. If you've ever used sites like JSFiddle to try out snippets of JS/HTML/CSS, this is basically the docker equivalent to try out Docker environments.

With PWD, I now have a dead simple way for anyone who wants to try out this viewer to spin up a demo MapGuide instance on PWD for themselves to check out the viewer.

Once you've proved to the site that you're are indeed a human and not a robot, you will enter the PWD console. From here, click + ADD NEW INSTANCE to start a new shell.



Then run the following commands to build the demo docker image and spin up the container

git clone https://github.com/jumpinjackie/mapguide-react-layout
cd mapguide-react-layout
./demo.sh

After a few minutes, you should see a port number appear beside the IP address



This is a link to the default Apache httpd page that is confirmation that the demo container is serving out web content to the outside world.



Now simply append /mapguide/index.php to that URL to access the demo landing page for this viewer. Pick any template on the list to load the viewer using that template.



You now have a live demo MapGuide Server with mapguide-react-layout (and the Sheboygan dataset) preloaded for you to play with to your heart's content for the next 4 hours, after which PWD will terminate your session and all the docker images/containers/etc that you created with it.

This was just one use case that I thought up in 5 minutes after discovering this awesome site! I'm sure there's plenty of other creative uses for such a site like this.

Many thanks to brucepc for his MGOS 3.1 docker image from which the demo image is based from.
Categories: OSGeo Planet

Jackie Ng: gRPC is very interesting

OSGeo Planet - Wed, 2017-03-22 16:19
MapGuide in its current form is a whole bucket of assorted libraries and technologies:
  • We use FDO for spatial data access
  • We use ACE (Adaptive Communication Environment) for:
    • Basic multi-threading primitives like mutexes, threads, etc
    • TCP/IP communication between the Web Tier and the Server Tier
    • Implementing a custom RPC layer on top of TCP/IP sockets. All of the service layer methods you use in the MapGuide API? They're all basically RPC calls sent over TCP/IP for the MapGuide Server to invoke its server-side eqvivalent. Most of the other classes that you pass into these service methods are essentially messages that are serialized/deserialized through the TCP/IP sockets. When you think about it, the MapGuide Web API is merely an RPC client for the MapGuide Server, which itself is an RPC server that does the actual work
  • We use Berkeley DBXML for the storage of all our XML-based resources
    • We have an object-oriented subset of these resource types (Feature Sources, Layer Definitions, Map Definitions, Symbol Definitions) in the MdfModel library with XML serialization/parsing code in the MdfParser library
    • Our Rendering and Stylization Engine work off of these MdfModel classes to render the maps that you see on your viewer
  • We use xerces for XML reading/writing XML in and out of DBXML
  • We use a custom modified (and somewhat ancient) version of SWIG to generate wrappers for our RPC client so that you can talk to the MapGuide Server in:
    • .net
    • Java
    • PHP
So why do I mention all of this?

I mention this, because I've recently been checking out gRPC, a cross-platform, cross-language RPC framework from Google.

And from what I've seen so far, gRPC could easily replace and simplify most of the technology stack we're currently using for MapGuide:
  • ACE? gRPC is the RPC framework! The only reason we'd keep ACE around would be for multi-threading facilities, but the C++ standard library at this point would be adequate enough to replace that as well
  • DBXML/MdfModel/xerces? gRPC is driven by Google Protocol Buffers.
    • Protobuf messages are strongly typed classes that serialize/deserialize into compact binary streams and is more efficient and faster than slinging around XML. Ever bemoan the fact you have to currently work with XML to manipulate maps/layers/etc? In .net you are reprieved if you use the Maestro API (where we provide strongly-typed classes for all the resource XML types), but for the other languages you have to figure out how to use the XML APIs/services provided by Java/PHP to work with the XML blobs that the MapGuide API gives and expects. With protobuf, you have none of these problems.
    • Protobuf messages can evolve in a backward-compatible manner
    • Because protobuf messages are already strongly-typed classes, it makes MdfModel/MdfParser redundant if you get the Rendering/Stylization engine to work against protobuf messages for maps/layers/symbols/styles/etc
    • If we ever wanted to add support for Mapbox Vector Tiles (which seems to be the de-facto vector tile format), well the spec is protobuf-based so ...
    • Protobuf would mean we no longer deal in XML, so we don't need Xerces for reading/writing XML and DBXML as the storage database (and all its cryptic error messages that can bubble up from the Resource Service APIs) can be replaced with something simpler. We may not even need a database at this point. Dumping protobuf messages to a structured file system could probably be a simpler solution
  • SWIG? gRPC and protobuf can already generate service stubs and protobuf message classes in the languages we currently target:
    • .net
    • Java
    • PHP
    • And if we wanted, we can also instantly generate a gRPC-based MapGuide API for:
      • node.js
      • Ruby
      • Python
      • C++
      • Android Java
      • Objective-C
      • Go
    • The best thing about this? All of this generated code is portable in their respective platforms and doesn't involve native code interop through "flattened" interfaces of C code wrapping the original C++ code, which is what SWIG ultimately does for any language we want to generate wrapper bindings out of. If it does involve native code interop, it's a concern that is taken care of by the respective gRPC/protobuf implementation for that language.
  • Combine a gRPC-based MapGuide Server with grpc-gateway and we'd have an instant REST API to easily build a client-side map viewer out of
  • gRPC works at a scale that is way beyond what we can currently achieve with MapGuide currently. After all, this is what Google uses themselves for building their various services
If what I said above doesn't make much sense, consider a practical example.

Say we had our Feature Service (which as a user of the MapGuide API, you should be familiar with) as a gRPC service Definition

// Message definitions for the request/response types below are omitted for brevity but basically every request and
// response type mentioned below will have eqvivalent protobuf message classes automatically generated along with
// the service

// Provides an abstraction layer for the storage and retrieval of feature data in a technology-independent way.
// The API lets you determine what storage technologies are available and what capabilities they have. Access
// to the storage technology is modeled as a connection. For example, you can connect to a file and do simple
// insertions or connect to a relational database and do transaction-based operations.
service FeatureService {
// Creates or updates a feature schema within the specified feature source.
// For this method to actually delete any schema elements, the matching elements
// in the input schema must be marked for deletion
rpc ApplySchema (ApplySchemaRequest) returns (BasicResponse);
rpc BeginTransaction (BeginTransactionRequest) returns (BeginTransactionResponse);
// Creates a feature source in the repository identified by the specified resource
// identifier, using the given feature source parameters.
rpc CreateFeatureSource (CreateFeatureSourceRequest) returns (BasicResponse);
rpc DeleteFeatures (DeleteFeaturesRequest) returns (DeleteFeaturesResponse);
// Gets the definitions of one or more schemas contained in the feature source for particular classes.
// If the specified schema name or a class name does not exist, this method will throw an exception.
rpc DescribeSchema (DescribeSchemaRequest) returns (DescribeSchemaResponse);
// This method enumerates all the providers and if they are FDO enabled for the specified provider and partial connection string.
rpc EnumerateDataStores (EnumerateDataStoresRequest) returns (EnumerateDataStoresResponse);
// Executes SQL statements NOT including SELECT statements.
rpc ExecuteSqlNonQuery (ExecuteSqlNonQueryRequest) returns (ExecuteSqlNonQueryResponse);
// Executes the SQL SELECT statement on the specified feature source.
rpc ExecuteSqlQuery (ExecuteSqlQueryRequest) returns (stream DataRecord);
// Gets the capabilities of an FDO Provider
rpc GetCapabilities (GetCapabilitiesRequest) returns (GetCapabilitiesResponse);
// Gets the class definition for the specified class
rpc GetClassDefinition (GetClassDefinitionRequest) returns (GetClassDefinitionResponse);
// Gets a list of the names of all classes available within a specified schema
rpc GetClasses (GetClassesRequest) returns (GetClassesResponse);
// Gets a set of connection values that are used to make connections to an FDO provider that permits multiple connections.
rpc GetConnectionPropertyValues (GetConnectionPropertyValuesRequest) returns (GetConnectionPropertyValuesResponse);
// Gets a list of the available FDO providers together with other information such as the names of the connection properties for each provider
rpc GetFeatureProviders (GetFeatureProvidersRequest) returns (GetFeatureProvidersResponse);
// Gets the locked features.
rpc GetLockedFeatures (GetLockedFeaturesRequest) returns (stream FeatureRecord);
// Gets all available long transactions for the provider
rpc GetLongTransactions (GetLongTransactionsRequest) returns (GetLongTransactionsResponse);
// This method returns all of the logical to physical schema mappings for the specified provider and partial connection string
rpc GetSchemaMapping (GetSchemaMappingRequest) returns (GetSchemaMappingResponse);
// Gets a list of the names of all of the schemas available in the feature source
rpc GetSchemas (GetSchemasRequest) returns (GetSchemasResponse);
// Gets all of the spatial contexts available in the feature source
rpc GetSpatialContexts (GetSpatialContextsRequest) returns (GetSpatialContextsResponse);
// Inserts a new feature into the specified feature class of the specified Feature Source
rpc InsertFeatures (InsertFeaturesRequest) returns (stream FeatureRecord);
// Selects groups of features from a feature source and applies filters to each of the groups according to the criteria set in the aggregate query option supplied
rpc SelectAggregate (SelectAggregateRequest) returns (stream DataRecord);
// Selects features from a feature source according to the criteria set in the query options provided
rpc SelectFeatures (SelectFeaturesRequest) returns (stream FeatureRecord);
// Set the active long transaction name for a feature source
rpc SetLongTransaction (SetLongTransactionRequest) returns (BasicResponse);
// Connects to the Feature Provider specified in the connection string
rpc TestConnection (TestConnectionRequest) returns (TestConnectionResponse);
// Executes commands contained in the given command set
rpc UpdateFeatures (UpdateFeaturesRequest) returns (UpdateFeaturesResponse);
// Updates all features that match the given filter with the specified property values
rpc UpdateMatchingFeatures (UpdateMatchingFeaturesRequest) returns (UpdateMatchingFeaturesResponse);
}

Running this service definition through the protoc compiler with grpc plugin gives us:
  • Auto-generated (and strongly-typed) protobuf classes for all the messages. ie: The request and response types for this service
  • An auto-generated FeatureService gRPC client ready to use in the language of our choice
  • An auto-generated gRPC server stub for FeatureService in the language of our choice ready for us to "fill in the blanks". For practical purposes, we'd generate this part in C++ and fill in the blanks by mapping the various service operations to their respective FDO APIs and its return values to our gRPC responses.
And at this point, we'd just need a simple C++ console program that bootstraps gRPC/FDO, registers our gRPC service implementation, start the gRPC server on a particular port and we'd have a functional Feature Service implementation in gRPC. Our auto-generated Feature Service client can connect to this host and port to immediately start talking to it.

The only real work is the "filling in the blanks" on the server part. Everything else is taken care of for us.

Extrapolate this to the rest of our services (Resource, Rendering, etc) and we basically have a gRPC-based MapGuide Server.

Also filling in the blanks is a conceptually simple exercise as well:
  • Feature Service - Pass down the APIs in FDO.
  • Rendering Service - Setup up FDO queries based on map/layers visible and pass query results to the Rendering/Stylization engine.
  • Resource Service - Read/write protobuf resources to some kind of persistent storage. It doesn't have to be something complex like DBXML, it can be as simple as a file system (that's what mg-desktop does for its resource service implementation btw)
  • Tile Service - It's just like the rendering service, but you're asking the Rendering/Stylization engine to render tile-sized content.
  • KML Service - Just like rendering service, but you're asking the Rendering/Stylization engine to render KML documents instead of images.
  • Drawing Service - Do we still care about DWF support? Well if we have to support this, it's just passing down to the APIs in DWF Toolkit.
  • Mapping Service - It's a mish-mash of tapping into the Rendering/Stylization engine and/or the DWF Toolkit.
  • Profiling Service - Just tap into whatever tracing/instrumentation APIs provided by gRPC.
Now because gRPC is cross-language, nothing says we have to use C++ for all the service implementations, it's just that most of the libraries and APIs we'd be mapping into are already in C++, so in practical terms we'd stick to the same language as well.

Front this with grpc-gateway, and we basically have our RESTful mapagent to build a map viewer against.

There's still a few unknowns:
  • How do we model file uploads/downloads?
  • Can server-side service implementations call other services?
Google's vast set of gPRC definitions for their various web services can answer the first part. The other part, will need more playing around.

The thought of a gRPC-based MapGuide Server is very exciting!
Categories: OSGeo Planet

gvSIG Team: gvSIG nominado en 3 categorías de los “Open Awards”. Abierta votación popular.

OSGeo Planet - Tue, 2017-03-21 15:45

El proyecto gvSIG vuelve a ver reconocida su propuesta de desarrollo de soluciones de geomática libre con la nominación a 3 categorías de los premios denominados “Open Awards”.

Tal y como se indica en la web de los premios, estos tienen como objetivo reconocer públicamente a empresas, administraciones, personalidades y comunidades que crean, apoyan y fomentan grandes soluciones con tecnologías Open Source y Software Libre.

Los Open Awards reconocen y premian los proyectos e iniciativas de código abierto que más han destacado durante el último año, impulsan la comunicación y la notoriedad pública de las empresas, proyectos y administraciones participantes en los premios y valoran el trabajo realizado por todos ellos.

Las 3 categorías en las que participa gvSIG reflejan en cierto modo la mirada amplia del proyecto, ya que se opta al “Mejor proveedor de servicios/soluciones” donde la Asociación gvSIG está demostrando que pueden ponerse en marcha nuevos modelos económicos de producción de software desde perspectivas colaborativas; a la “Plataforma/proyecto más innovador” una categoría que se centra en la parte técnico-científica del proyecto y donde gvSIG Suite se constituye como la plataforma integral para solventar las necesidades “geo” de cualquier organización, con aplicaciones de escritorio, móviles y web siguiendo la filosofía de las Infraestructuras de Datos Espaciales; por último la categoría donde más ilusión nos hace participar y que en realidad es un reconocimiento a todos vosotros, los que día a día, desde vuestro rincón apoyáis y ayudáis a gvSIG, la nominación a “Mejor comunidad tecnológica”.

En estos premios hay una primera fase de votaciones públicas de la que saldrá un “top cinco”. A partir de ahí un jurado determinará los ganadores de cada categoría.

¿Qué pasos hay que seguir para votad?

  1. Votad en las categorías que consideres (desde hoy hasta el 30 de abril).
  2. Confirmad voto por email.

Como aliciente adicional, por votar se tiene acceso exclusivo al eBook “OpenExpo Tendencias Open Source y Software Libre 2017”.

Los enlaces directos a las categorías donde está gvSIG nominado son:

Desde ya os agradecemos a todos vosotros vuestro apoyo y tened seguro que nosotros también os votaremos como “Mejor comunidad tecnológica”.

Categories: OSGeo Planet

GeoSolutions: GeoServer Code Sprint needs you

OSGeo Planet - Tue, 2017-03-21 12:22

GeoServer

Dear Reader,

everything is ready in GeoSolutions for next week's GeoServer code sprint which will take place in our offices during the week of March 27th.

[caption id="attachment_3380" align="aligncenter" width="531"]Sprint 2017 Sprint 2017[/caption]

The main focus will be on refactoring GeoServer's REST API towards a more modern approach (se this page for some insights). A number of GeoServer developers from various organizations will gather from all over world for this work and your support in funding this initiative would help us out with the expenses, therefore, we are asking help to all our readers. Sponsorship opportunities are available for you to contribute on the OSGeo wiki.

Happy sprinting to everybody! The GeoSolutions team,
Categories: OSGeo Planet

gvSIG Team: Aprende los secretos del geoprocesamiento vectorial en gvSIG con este vídeo-tutorial

OSGeo Planet - Tue, 2017-03-21 09:01

En gvSIG Desktop disponemos de más de 350 geoprocesos, esto sin contar con plugins como el recientemente anunciado de Jgrass. Un buen porcentaje de esos geoprocesos se aplican sobre capas vectoriales, desde los más comunes -como el área de influencia, cortar, unir,…- a otros más específicos y menos conocidos.
Hoy os presentamos un vídeo-tutorial en el que en pocos minutos aprenderéis el funcionamiento de los geoprocesos de gvSIG, mediante una serie de ejercicios prácticos que nos permitan comprender la sencillez con la que podemos utilizar los algoritmos disponibles en la aplicación.
En la parte final del vídeo-tutorial podréis aprender a manejar el modelador de geoprocesos; una herramienta muy útil y no muy conocida por los usuarios de gvSIG.
Seguid leyendo…


Filed under: gvSIG Desktop, spanish Tagged: geoprocesamiento, geoprocesos, modelador
Categories: OSGeo Planet

GeoSolutions: New release of MapStore 2 with theming support

OSGeo Planet - Mon, 2017-03-20 11:03

blog

Dear Reader,

we are pleased to announce a new release of MapStore 2, our flagship Open Source webgis product, which we have called 2017.02.00. The full list of changes for this release can be found here, but let us now concentrate on the latest most interesting additions.

Advanced Theming The main Feature of this release is the possibility to have different themes, as shown in the Gallery below. [gallery type="slideshow" ids="3319,3320,3324,3325,3326,3321,3327,3328,3329,3330"]

MapStore 2 was conceived with the goal to be highly customizable, therefore we have worked hard on the look and feel from the beginning to create a product that would easily adapt to predefined graphical guidelines as well as a framework which could be easy to integrate with 3rd party applications.

With this release the goal has been achieved. We have refactored the original theme simplifying greatly the steps to create new themes and switch between them. You can try to switch it live from the home page, there is a specific combo with some predefined styles.

On the technical side, we have refactored MapStore 2 theme support using less, hence now creating your own theme to match your company's visual design guidelines is very simple. We are developing an example that allows you to customize your theme directly from the web page, you can see it here below or test it here.

[gallery type="slideshow" ids="3333,3334"] Balloon Tutorial The balloon tutorial is now ready also with html support. You can try it live by clicking on "Tutorial" in the map's burger menu, as an instance in this map. [gallery type="slideshow" ids="3336,3337,3338,3339,3340"]   Notes for Developers

This release has a number of changes that are crucial to know for the developers since they break compatibility with older versions. Here, you can find the details of what we updated and how to migrate your application.

We strongly believe that these changes will speed up the development and improve the quality and the readability of the code (particularly redux-observable). If you will find yourself struggling with these changes, reach out for us on the developer mailing list and we will help you out.

We also looked around for a tool to produce developers' docs that would satisfy our needs in the longer term.  We found in docma a great tool as it allows us to provide both generic guides as well as to document our components, plugins and JavaScript API inline using jsDoc + Markdown. You can find the current version of the developer documentation here. Twitter Account

MapStore 2 has now its own twitter account which it is using to let us know how it feels as well as to share useful information and insights.

What we are working on for the next release

The main focus for the next release is the implementation of a JavaScript API to allow you to include MapStore 2 in your application or web-site and interact with it in more advanced ways than a simple IFRAME. We are also going to focus on the following items:

  • Improve developer's documentation
  • Improve the management of Maps, in order to allow users to manage them also from the map itself
  • Better interaction with WFS

In the longer term, we have a number of features and functionalities in our plans like editing, advanced templating, styling, OAUTH 2.0, and more…

So, Stay tuned and happy webmapping!

If you are interested in learning about how we can help you achieving your goals with open source products like GeoServerMapstore, GeoNode and GeoNetwork through our Enterprise Support Services and GeoServer Deployment Warranty offerings, feel free to contact us!

The GeoSolutions team,
Categories: OSGeo Planet

gvSIG Team: Disponible vídeo-tutorial para aprender Geoestadística con gvSIG

OSGeo Planet - Wed, 2017-03-15 09:06

El paquete estadístico R es uno de los más flexibles, potentes y profesionales que existen actualmente para realizar tareas estadísticas de todo tipo, desde las más elementales, hasta las más avanzadas. Y, lo más importante, es software libre.

Desde sus últimas versiones, gvSIG Desktop ha incluido plugins para integrar R, abriendo así la posibilidad a realizar todo tipo de análisis geoestadísticos.

Geólogos, biólogos, ecólogos, agrónomos, ingenieros, meteorólogos, sociólogos…por nombrar sólo unos pocos profesionales, requieren del estudio de información estadístico de información georreferenciada.

Desde la Asociación gvSIG os presentamos un vídeo-tutorial que os permitirá introduciros en el funcionamiento de la dupla gvSIG-R.

Si hemos despertado tú interés, sigue leyendo…


Filed under: gvSIG Desktop, spanish Tagged: Geoestadística, r
Categories: OSGeo Planet

GeoServer Team: GeoServer 2.11-RC1 Released

OSGeo Planet - Tue, 2017-03-14 07:23

We are happy to announce the release of GeoServer 2.11-RC1. Downloads are available (zipwardmg and exe) along with docs and extensions.

This is a release candidate of GeoServer not intended for production use. This release is made in conjunction with GeoTools 16-RC1 and GeoWebCache 1.11-RC1.

Thanks to everyone who provided feedback, bug reports and fixes. Here are some of the changes included in 2.11-RC1:

  • Incompatibilities with GeoFence and Control-flow have been resolved
  • Empty WFS Transactions (which perform no changes) no indicating everything has changed
  • Improvements to WFS GetFeature support for 3D BBOX requests
  • We have one known regression with the windows service installer (memory setting is incorrect)
  • Please additional details see the release notes (2.11-RC12.11-beta )
Release Candidate Testing

The 2.11 release is expected in March, this release candidate is a “dry run” where we confirm new functionality is working as expected and double check the packaging and release process.

Please note that GeoServer 2.9 has reached its end-0f-life. If your organization has not yet upgraded please give us hand by evaluating 2.11-RC1 and providing feedback and your experiences for the development team. This is a win/win situation where your participation can both assist the GeoServer team and reduce your risk when upgrading.

Corrected default AnchorPoint for LabelPlacement

An issue with SLD 1.0 rendering has been fixed – when a LabelPlacement did not include a AnchorPoint we were using the wrong default!

  • BEFORE: default anchor point was X=0.5 and Y=0.5 – which is at the middle height and middle length of the label text.
  • AFTER: default anchor point was X=0.0 and Y=0.5 – which is at the middle height of the lefthand
    side of the label text.

This is a long standing issue that was only just noticed in February. If you need to “restore” the incorrect behaviour please startup with -Dorg.geotools.renderer.style.legacyAnchorPoint=true system property.

Startup Performance

With extensive improvements to startup performance and OGC requests for large installations we are looking forward to feedback from your experience testing.

About GeoServer 2.11

GeoServer 2.11 is scheduled for March 2017 release. This puts GeoServer back on our six month “time boxed” release schedule.

  • OAuth2 for GeoServer (GeoSolutions)
  • YSLD has graduated and is now available for download as a supported extension
  • Vector tiles has graduate and is now available for download as an extension
  • The rendering engine continues to improve with underlying labels now available as a vendor option
  • A new “opaque container” layer group mode can be used to publish a basemap while completely restricting access to the individual layers.
  • Layer group security restrictions are now available
  • Latest in performance optimizations in GeoServer (GeoSolutions)
  • Improved lookup of EPSG codes allows GeoServer to automatically match EPSG codes making shapefiles easier to import into a database (or publish individually).
Categories: OSGeo Planet

Even Rouault: Dealing with huge vector GeoPackage databases in GDAL/OGR

OSGeo Planet - Sat, 2017-03-11 22:25
Recently, I've fixed a bug in the OGR OpenFileGDB driver, the driver made from the reverse engineering the ESRI FileGeoDatabase format, so as to be able to read tables whose section that enumerates and describes fields is located beyond the first 4 GB of the file. This table from the 2016 TIGER database is indeed featuring all linear edges of the USA and is 15 GB large (feature and spatial indexes included), with 85 million features.

Some time before, I had to deal with a smaller database - 1.7 GB as GeoPackage - with 5.4 million polygons (bounding box) from the cadastre of an Italian province. One issue I noticed is that when you want to get the summary of the layer, with ogrinfo -al -so the.gpkg, it was very slow. The reason is that this summary includes the feature count, and there's no way to get this metadata quickly, apart from running the "SELECT COUNT(*) FROM the_table" request, which causes a full scan of the table. For small databases, this runs fast, but when going into the gigabyte realm, this can take several dozains of seconds. But getting the spatial extent of the layer, which is one of the other information displayed by the summary mode of ogrinfo, is fast since the gpkg_contents "system" table of a GeoPackage database includes the bounding box of the table. So my idea was to extend the definition of the gpkg_contents table to add a new column, ogr_feature_count, to store the feature count. I went to implement that, and it worked fine. The synchronization of the value of ogr_feature_count after edits can be done with 2 SQLite triggers, on row insertion and deletion, and that  works with implementations that are not aware of the existence of this new column. Like older OGR versions. Unfortunately it appears that at least one other implementation completely rejected such databases. There is some inconsistency in the GeoPackage specification if additional columns are accepted or not in system tables. From the /base/core/contents/data/table_def test case, "Column order, check constraint and trigger definitions, and other column definitions in the returned sql are irrelevant.", it would seem that additional columns should still be considered as a valid GeoPackage. Anyway, that's only the theory and we don't want to break interoperability for just a nice-to-have feature... So I went to change the design a bit and created a new table, gpkg_ogr_contents, with a table_name and feature_count columns. I'm aware that I should not borrow the gpkg_ prefix, but I felt it was safer to do so since other implementations will probably ignore any unknown gpkg_ prefixed table. And the addition of the ogr_ prefix makes collisions with future extension of the GeoPackage specification unlikely. The content of this table is also maintained in synchronization with the data table thanks to two triggers, and this makes the other software that rejected my first attempt happy. Problem solved.

Let's come back to our 13 GB FileGeoDatabase. My first attempt to convert is to GeoPackage with ogr2ogr resulted in converting the features in about half an hour, but once this 100% stage reached, the finalization, which includes building the spatial index took ages. So long, that after a whole night it wasn't yet finished and seriously making the computer non responsive due to massive I/O activity. In the GeoPackage driver, the spatial index is indeed created after feature insertion, so that the feature table and spatial index tables are well separated in the file, and from previous experiment with the Spatialite driver, it proved to be the right strategy. Populating the SQLite R-Tree is done with a simple statement: INSERT INTO my_rtree SELECT fid, ST_MinX(geom), ST_MaxX(geom), ST_MinY(geom), ST_MaxY(geom) FROM the_table. Analyzing what happens in the SQLite code is not easy when you are not familiar with that code base, but my intuition is that there was constant back and forth between the geometry data area and the RTree area in the file, making the SQLite page cache inefficient. So I decided to experiment with a more progressive approach. Let's iterate over the feature table and collect the fid, minx, max, miny, maxy by chunks of 100 000 rows, and the insert those 100 000 bounding boxes into the R-Tree, and loop again unil we have completely read the feature table. With such a strategy, the spatial index can now be built in 4h30. The resulting GeoPackage file weights 31.6 GB, so twice at large than the FileGeoDatabase. One of the reasons for the difference must be due to geometries in FileGeoDatabase being compressed (quantization for coordinate precision, delta encoding and use of variable integer) whereas GeoPackage uses a uncompressed SQLite BLOB based on OGC WKB.
My first attempt at opening it in QGIS resulted in the UI to be frozen, probably for hours. The reason is that QGIS always issues a spatial filter, even when requesting on a area of interest that is at least as large as the extent of the layer, where there is no performance gain to expect from using it. So the first optimization was in the OGR GeoPackage to detect that situation and to not translate the OGR spatial filter as SQLite R-Tree filter. QGIS could now open the database and progressively displays the features. Unfortunately when zooming in, the UI became frozen again. When applying a spatial filter, the GeoPackage driver created a SQL request like the following one:
SELECT * FROM the_table WHERE fid IN        (SELECT id FROM the_rtree WHERE         xmin <= bbox_xmax AND xmax >= bbox_xmin AND        ymin <= bboy_ymay AND ymay >= bboy_ymin)
It turns out that the sub-select (the one that fetches the feature ID from the spatial index) is apparently entirely run before the outer select (the one that returns geometry and attributes) starts being evaluated. This way of expressing the spatial filter came from the Spatialite driver (since GeoPackage and Spatialite use the exact same mechanisms for spatial indexing), itself based on examples from an old Spatialite tutorial. For not too big databases, this runs well. After some experiment, it turns out that doing a JOIN between the feature table and the RTree virtual table makes it possible to have a non blocking request:
SELECT * FROM the_table t JOIN the_rtree r ON t.fid = r.idWHERE r.xmin <= bbox_xmax AND r.xmax >= bbox_xmin AND      r.ymin <= bboy_ymax AND r.ymax >= bboy_ymin
Now QGIS is completely responsive, although I find that even on high zoom levels, the performance is somehow disappointing, ie features appear rather slowly. There seems to be some threshold effect on the size of the database, since the performance is rather good on the Italian province cadastral use case.

Another experiment showed that increasing the SQLite page size from 1024 bytes (the default in SQLite 3.11 or earlier) to 4096 bytes (the default since SQLite 3.12) decreases the database size to 28.8 GB. This new page size of 4096 bytes is now used by default by the OGR SQLite and GPKG drivers (unless OGR_SQLITE_PRAGMA=page_size=xxxx is specified as a configuration option).

I also discovered that increasing the SQLite page cache from its 2 MB default to 2 GB (with --config OGR_SQLITE_CACHE 2000) significantly improved the time to build the spatial index, decreasing the total conversion time from 4h30 to 2h10. 2GB is just a value selected at random. It might be too large or perhaps a larger value would help further.

All improvements mentionned above (faster spatial index creation, better use of spatial index and change of default page size) are now in GDAL trunk, and will be available in the upcoming 2.2.0 release.
Categories: OSGeo Planet
Syndicate content