A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant


Lidar – Accuracy Versus Resolution

Digital Coast GeoZone

[Update: In November 2014, ASPRS published new guidance for accuracy classes that includes recommendations for pulse spacing in addition to vertical accuracy.]

Resolution != Accuracy

It seems like every day I hear a statement about high-resolution lidar that bugs the heck out of me. I even hear it from our staff at the Center. So, I thought I’d write a little entry about it and see if any of them pick up on it. So, what I hear is, “We need high resolution lidar for X, Y, and Z.” Sometimes it’s for estimating sea level change impacts, sometimes it’s for habitat modeling; could be for nearly anything. While I do often hear it for sea level change, my example is wetlands delineation in relatively flat areas like South Carolina or Florida. To quote one friend (we’ll call him Randy) interested in wetlands:

It appears that most of it will be point spacing…

View original post 760 more words

The end is nigh…

I read a blog post a few weeks ago about the inevitable demise of ArcMap. When ArcGIS Pro launched a couple of years ago, I immediately started preparing for the end of ArcMap. By that, I mean I played around with Pro for a few weeks then put it away until the corporate overlords decided it was time to switch.

I had to sign up for a free trial the other day for something, and found this:

What happened to ArcMap?

After logging in and heading to the downloads, I found this:

What happened to ArcMap?!?!?!!

The forums pointed it out as well. I’ve been keeping a side eye on ArcGIS Pro development over the last few years, so I’m starting the transition from October, with the aim of using it as my daily driver by December. I’ve just started training our junior consultant who comes from a CAD background on GIS as well, so I may just start him out on ArcGIS Pro from the jump.

Thoughts on spatial data warehousing

I’ve been thinking quite a bit lately about how to store spatial data. It’s something I’ve covered here and my attitude towards this topic has evolved over the years.

The organisations I’ve worked in have mountains of spatial data accumulated over the years. The data is stored in shapefiles, geodatabases, normal databases, spreadsheets, documents, reports, photos…Why is it this way? It doesn’t have to be this way. It shouldn’t be this way!

In the course of my research for a topic for my project for next year, I’ve honed in on the methods for implementing an enterprise geoportal within an existing spatial data infrastructure. However, I feel like my focus is shifting to the data that the geoportal is trying to expose to a larger audience.

The concepts of a spatial data warehouse and a spatially enabled operational data store have been intriguing me. A regular GIS task involves comparing spatial data across a time period, analysing trends and presenting the results in a map or report. Why aren’t we storing this historical data in a SDW that’s optimised for reporting?

Non-spatial data can come from a variety of sources as well – spreadsheets, other databases etc. Another common GIS task is to spatially enable these datasets. Why are we not storing the outputs in a spatial enabled operational data store in an open format like GML?

I think it’s because to plan and implement a SDW/S-ODS takes time (and money). With a normal EDW, the organisation will not need much convincing to see the benefit of implementing one. “Spatial” is still seen as an “add-on”, or a “nice-to-have”.

The issue with names

I recently underwent a name change, and though I have yet to make it official (who wants to waste a Saturday at Home Affairs?), I have been thinking about the implications of my name change.

Now that my surname contains a hyphen and is 18 characters long (with my full name now 26 characters), I’ve been wondering how I should abbreviate it. Some hasty Googling shows that there is no standard for this. My entire life I just assumed that the first part of the surname takes precedence so the initials remain the same. In other words, Cindy Lee Williams (CLW) becomes Cindy Lee Williams-Jayakumar (CLW).

I toyed around with the idea of dropping my middle name (itself having been an issue with people assuming I’m Cindy-Lee and not Cindy Lee) to become Cindy Williams-Jayakumar (CW), but the thought of having only two initials terrified me.

I came across this blog post which calls out the assumptions programmers make when building systems which need to accept names (I’m guessing that’s about 95% of all systems). Now that my name has become slightly more complicated, I’m going to be more aware of my own assumptions when writing code, and not just when it comes to validating names.

I’ve also decided to be a bit more difficult and use CWJ as my initials. I had CLW for 27 years, it was time for a change.

Database access via Python

In my ongoing quest to do absolutely everything through Python, I’ve been looking a lot lately at manipulating databases. I’ve been using arcpy to access GIS databases for years, and last year I finally got around to using pyodbc (and pypyodbc) for accessing SQL Server databases.

Now that I’m in an Oracle environment, Oracle has provided the cx_Oracle library to directly connect to databases. I have yet to test that though. What I’m interested in at the moment is creating and accessing databases for personal use.

I considered MongoDB for a while, but I don’t think I want to go NoSQL yet. This is why I have been experimenting with SQLite (through the sqlite3 library), as it is included in the Python install, and has the delightful SpatiaLite extension. The slogan goes against my one of my mottos (Spatial is Special) while supporting my other motto (Everything is Spatial).