Leaving the GIS industry: bye for now

The GIS industry’s loss is software engineering’s gain. All the best Alex! Reblogging to highlight an excellent farewell post.

Alex Tereshenkov

After working for more than 10 years in geospatial industry, I have decided to change the field and focus on pure software engineering. I have quite enjoyed using GIS tools to solve various computational problems and software development tools to build useful GIS products and services for my customers. However, as my interests starting shifting towards clean code and programming per se – I have noticed I was reading in the bed Refactoring by Martin Fowler way more often than The ESRI guide to GIS analysis by Andy Mitchell – so I thought it would be useful to try changing the career path.

As I won’t be using any of the GIS software any longer, I won’t be able to post any new practical material that would be useful to GIS analysts and developers. However, in this post I would like to share some of the last thoughts I have…

View original post 2,177 more words

Advertisements

A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant

Lidar – Accuracy Versus Resolution

Digital Coast GeoZone

[Update: In November 2014, ASPRS published new guidance for accuracy classes that includes recommendations for pulse spacing in addition to vertical accuracy.]

Resolution != Accuracy

It seems like every day I hear a statement about high-resolution lidar that bugs the heck out of me. I even hear it from our staff at the Center. So, I thought I’d write a little entry about it and see if any of them pick up on it. So, what I hear is, “We need high resolution lidar for X, Y, and Z.” Sometimes it’s for estimating sea level change impacts, sometimes it’s for habitat modeling; could be for nearly anything. While I do often hear it for sea level change, my example is wetlands delineation in relatively flat areas like South Carolina or Florida. To quote one friend (we’ll call him Randy) interested in wetlands:

It appears that most of it will be point spacing…

View original post 760 more words

The end is nigh…

I read a blog post a few weeks ago about the inevitable demise of ArcMap. When ArcGIS Pro launched a couple of years ago, I immediately started preparing for the end of ArcMap. By that, I mean I played around with Pro for a few weeks then put it away until the corporate overlords decided it was time to switch.

I had to sign up for a free trial the other day for something, and found this:

What happened to ArcMap?

After logging in and heading to the downloads, I found this:

What happened to ArcMap?!?!?!!

The forums pointed it out as well. I’ve been keeping a side eye on ArcGIS Pro development over the last few years, so I’m starting the transition from October, with the aim of using it as my daily driver by December. I’ve just started training our junior consultant who comes from a CAD background on GIS as well, so I may just start him out on ArcGIS Pro from the jump.

The search for the ultimate productivity system continues

I spend far too much time thinking about systems. I have a system for everything – my email accounts, my household, my work, my studies and for how all those systems interact. Sure, everyone has a system for those, you might say. For me though, it is my preferred form of procrastination, in the sense that I have a tangible result (a new system!) at the end of it, as opposed to just whiling away the time on YouTube.

Let’s take a look at the evolution of my productivity system. I started out with OneNote in 2011, integrating it with Outlook Tasks in 2012. That plodded along for a bit, until I realised that tracking tasks in Outlook is ridiculous and I needed something web-based. I fiddled around a bit with some to-do list managers and settled on Wunderlist. I kept trying to link it with OneNote, but that turned out to be a useless fight.

I then started messing around with Trello, essentially duplicating what I was doing in Wunderlist. I waffled between the two until Microsoft bought Wunderlist, and now created To-Do, which will eventually replace Wunderlist but has very few features at the time of writing. There’s also Planner, which is the Office 365 version of a combination of Trello and Jira I guess? I don’t know.

I feel like I’m in limbo. At work, I’m currently using Planner with the team to track project tasks, linked to the relevant Notebooks, Groups and SharePoint libraries (accessible through the OneDrive mobile apps). I did link Wunderlist to Planner, but that just seemed weird.

I’m still using Wunderlist for daily task reminders, bill reminders, shopping list and food prep. I’m using Trello to track the food inventory of my household. I’m (kind of) using the normal Outlook calendar to track appointments. In other words, my system is a mess and I need to throw everything out and start over!

Thoughts on spatial data warehousing

I’ve been thinking quite a bit lately about how to store spatial data. It’s something I’ve covered here and my attitude towards this topic has evolved over the years.

The organisations I’ve worked in have mountains of spatial data accumulated over the years. The data is stored in shapefiles, geodatabases, normal databases, spreadsheets, documents, reports, photos…Why is it this way? It doesn’t have to be this way. It shouldn’t be this way!

In the course of my research for a topic for my project for next year, I’ve honed in on the methods for implementing an enterprise geoportal within an existing spatial data infrastructure. However, I feel like my focus is shifting to the data that the geoportal is trying to expose to a larger audience.

The concepts of a spatial data warehouse and a spatially enabled operational data store have been intriguing me. A regular GIS task involves comparing spatial data across a time period, analysing trends and presenting the results in a map or report. Why aren’t we storing this historical data in a SDW that’s optimised for reporting?

Non-spatial data can come from a variety of sources as well – spreadsheets, other databases etc. Another common GIS task is to spatially enable these datasets. Why are we not storing the outputs in a spatial enabled operational data store in an open format like GML?

I think it’s because to plan and implement a SDW/S-ODS takes time (and money). With a normal EDW, the organisation will not need much convincing to see the benefit of implementing one. “Spatial” is still seen as an “add-on”, or a “nice-to-have”.