Thoughts on spatial data warehousing

I’ve been thinking quite a bit lately about how to store spatial data. It’s something I’ve covered here and my attitude towards this topic has evolved over the years.

The organisations I’ve worked in have mountains of spatial data accumulated over the years. The data is stored in shapefiles, geodatabases, normal databases, spreadsheets, documents, reports, photos…Why is it this way? It doesn’t have to be this way. It shouldn’t be this way!

In the course of my research for a topic for my project for next year, I’ve honed in on the methods for implementing an enterprise geoportal within an existing spatial data infrastructure. However, I feel like my focus is shifting to the data that the geoportal is trying to expose to a larger audience.

The concepts of a spatial data warehouse and a spatially enabled operational data store have been intriguing me. A regular GIS task involves comparing spatial data across a time period, analysing trends and presenting the results in a map or report. Why aren’t we storing this historical data in a SDW that’s optimised for reporting?

Non-spatial data can come from a variety of sources as well – spreadsheets, other databases etc. Another common GIS task is to spatially enable these datasets. Why are we not storing the outputs in a spatial enabled operational data store in an open format like GML?

I think it’s because to plan and implement a SDW/S-ODS takes time (and money). With a normal EDW, the organisation will not need much convincing to see the benefit of implementing one. “Spatial” is still seen as an “add-on”, or a “nice-to-have”.

Is 2016 the year of location?

I came across an interesting post on The Next Web the other day. It asks if 2016 is the year of location. The premise of this blog is all about that – the fact that everything we do is tied to a location. Everything is spatial.

I have believed for quite a while now that as GIS grows more and more, and becomes more integrated into every domain, that the role of the GIS professional will shift drastically. The “pure” GIS professional will be enterprise/programming focussed, supporting the domain GIS users who aren’t necessarily trained GIS users, but merely use GIS as part of their toolset.

ArcMap Woes Part 6: Random failure of gp tools on grouped layers

I had a series of errors recently while I was trying to run a few geoprocessing tools manually on layers within a group layer. The error message kept saying it couldn’t open the layer, even though the layer was clearly in the map. Even restarting ArcMap did not help.

I eventually discovered that geoprocessing tools may randomly fail on layers which are part of a group layer. I say randomly because sometimes the tools would run as expected, other times not. I’ve only experienced this anomaly when using the ArcMap interface – running the same sequence of tools on all the layers in a group layer in a for loop in Python does not give this error.

Geodesign: Change Your World MOOC: I think I’ll keep my world the same for now.

I was so excited to see that Penn State was offering another MOOC this year, this time about geodesign. It’s been on my watchlist for the past few months, so when I got the email to announce that class was starting, I immediately checked it out.

After my last Penn State MOOC though, I was a bit more wary this time around. A cursory glance down the left pane showed me what I was hoping I would not find:Geodesign_PeerAssessments

I half-heartedly went through the course content after that, but I had already decided not to carry on. The peer assessment assignment is 40% of the grade for the course. I’ve mentioned here before that peer assessments in a MOOC are a dealbreaker for me. I learn best on my own. I enjoy working as part of a team, but on my own as part of a team.

This week I started with the two other modules I need to get into Hons. The one is about software engineering principles, while the other is about advanced database development using an old version of Oracle (why, Unisa, why). I’m quite excited about the SQL one, as it will give me more opportunities to flex my SQL muscles, and to get exposure to a different database platform (something other than SQL Server).

Basically, even if I wanted to do this MOOC now, I can’t. Best of luck to the Geodesign students, and hopefully with the next offering, they will relax the peer assessment requirement.

ArcMap Woes Part 5: Exporting map without options

Sometimes, when in ArcMap, and one wants to export a map as a JPG or PNG or something, there won’t be any options. You can’t change the resolution, can’t change the export quality, can’t include a world file or anything. This used to frustrate me enormously, because then I would have to restart ArcMap to get the options to appear.

Sometimes even that wouldn’t work. I would then have to disturb a colleague and have them open the same mxd, to find that the options have miraculously reappeared. I have since figured out that if this happens, one needs to cancel the export dialog, switch to layout view and then export the map.

I am aware that it is possible to export the map with options while in data view; it’s just a random thing that happens. At least I know how to solve this particular issue.

ArcMap Woes Part 4: Permissions errors

Sometimes when I’m in ArcMap, I can’t start an edit session. Normally, I restart ArcMap and everything is fine.

Recently, for about a week, I could not edit inside any file gdb, or create new feature classes inside of an existing one or a new one. The gdb was locking itself when I was in an mxd that was accessing the feature classes inside it.

No other application was accessing it at the time i.e. ArcCatalog was closed. I would reboot the VM, open a blank map document, drag in the feature class I wanted, then try to edit that feature class, or create a new feature class in that gdb. No such luck.

I mentioned that this happened for about a week. After that, everything was fine again. I’m not sure why. Windows Updates have broken file gdbs before, but I don’t think it was that.

I think that IT changed some group policies and that affected my host where the data is stored, which in turn affected the VM. I know that sounds crazy, but I’m used to accepting crazy by now when it comes to Arc. It’s still not enough to make me go to QGIS though.

ArcMap Woes Part 3: Mxds that get tired

I tend to refresh my map documents every 3 days or so. By that, I mean I drag all my layers into a blank mxd, and save that as my new working document.

After a round of analysis and editing (the length of the round varies), the current mxd seems to get “tired”. ArcMap takes a few seconds longer to respond if I make a change, open a tool or do anything. Closing that mxd and opening a different one results in ArcMap functioning as expected; try opening that first mxd again, and it’s back to slowness and frustration.

This tends to happen particularly with mxds containing CAD drawings, or ones I’ve used to perform a lot of edits in. I’m not sure why it happens, all I know is that if ArcMap starts “getting slow”, I open a blank mxd and carry on. I don’t fight it, I just accept it for what it is.

Now that ArcGIS Pro is doing rounds, I’m hoping that this won’t be an issue anymore.

This doesn’t bother me as much as it did before, as I do most of my work/processing through Python now anyway. If I really need to, I open ArcCatalog to look at something.

The problem gets a bit more serious when geoprocessing tools start failing randomly, and you spend three hours checking your inputs, modifying parameters and pulling your hair out before you remember to open a blank document, run the same tool in there and get the results you were expecting.