A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant


Thoughts on spatial data warehousing

I’ve been thinking quite a bit lately about how to store spatial data. It’s something I’ve covered here and my attitude towards this topic has evolved over the years.

The organisations I’ve worked in have mountains of spatial data accumulated over the years. The data is stored in shapefiles, geodatabases, normal databases, spreadsheets, documents, reports, photos…Why is it this way? It doesn’t have to be this way. It shouldn’t be this way!

In the course of my research for a topic for my project for next year, I’ve honed in on the methods for implementing an enterprise geoportal within an existing spatial data infrastructure. However, I feel like my focus is shifting to the data that the geoportal is trying to expose to a larger audience.

The concepts of a spatial data warehouse and a spatially enabled operational data store have been intriguing me. A regular GIS task involves comparing spatial data across a time period, analysing trends and presenting the results in a map or report. Why aren’t we storing this historical data in a SDW that’s optimised for reporting?

Non-spatial data can come from a variety of sources as well – spreadsheets, other databases etc. Another common GIS task is to spatially enable these datasets. Why are we not storing the outputs in a spatial enabled operational data store in an open format like GML?

I think it’s because to plan and implement a SDW/S-ODS takes time (and money). With a normal EDW, the organisation will not need much convincing to see the benefit of implementing one. “Spatial” is still seen as an “add-on”, or a “nice-to-have”.

Is 2016 the year of location?

I came across an interesting post on The Next Web the other day. It asks if 2016 is the year of location. The premise of this blog is all about that – the fact that everything we do is tied to a location. Everything is spatial.

I have believed for quite a while now that as GIS grows more and more, and becomes more integrated into every domain, that the role of the GIS professional will shift drastically. The “pure” GIS professional will be enterprise/programming focussed, supporting the domain GIS users who aren’t necessarily trained GIS users, but merely use GIS as part of their toolset.

ArcMap Woes Part 6: Random failure of gp tools on grouped layers

I had a series of errors recently while I was trying to run a few geoprocessing tools manually on layers within a group layer. The error message kept saying it couldn’t open the layer, even though the layer was clearly in the map. Even restarting ArcMap did not help.

I eventually discovered that geoprocessing tools may randomly fail on layers which are part of a group layer. I say randomly because sometimes the tools would run as expected, other times not. I’ve only experienced this anomaly when using the ArcMap interface – running the same sequence of tools on all the layers in a group layer in a for loop in Python does not give this error.

Geodesign: Change Your World MOOC: I think I’ll keep my world the same for now.

I was so excited to see that Penn State was offering another MOOC this year, this time about geodesign. It’s been on my watchlist for the past few months, so when I got the email to announce that class was starting, I immediately checked it out.

After my last Penn State MOOC though, I was a bit more wary this time around. A cursory glance down the left pane showed me what I was hoping I would not find:Geodesign_PeerAssessments

I half-heartedly went through the course content after that, but I had already decided not to carry on. The peer assessment assignment is 40% of the grade for the course. I’ve mentioned here before that peer assessments in a MOOC are a dealbreaker for me. I learn best on my own. I enjoy working as part of a team, but on my own as part of a team.

This week I started with the two other modules I need to get into Hons. The one is about software engineering principles, while the other is about advanced database development using an old version of Oracle (why, Unisa, why). I’m quite excited about the SQL one, as it will give me more opportunities to flex my SQL muscles, and to get exposure to a different database platform (something other than SQL Server).

Basically, even if I wanted to do this MOOC now, I can’t. Best of luck to the Geodesign students, and hopefully with the next offering, they will relax the peer assessment requirement.

ArcMap Woes Part 5: Exporting map without options

Sometimes, when in ArcMap, and one wants to export a map as a JPG or PNG or something, there won’t be any options. You can’t change the resolution, can’t change the export quality, can’t include a world file or anything. This used to frustrate me enormously, because then I would have to restart ArcMap to get the options to appear.

Sometimes even that wouldn’t work. I would then have to disturb a colleague and have them open the same mxd, to find that the options have miraculously reappeared. I have since figured out that if this happens, one needs to cancel the export dialog, switch to layout view and then export the map.

I am aware that it is possible to export the map with options while in data view; it’s just a random thing that happens. At least I know how to solve this particular issue.

ArcMap Woes Part 4: Permissions errors

Sometimes when I’m in ArcMap, I can’t start an edit session. Normally, I restart ArcMap and everything is fine.

Recently, for about a week, I could not edit inside any file gdb, or create new feature classes inside of an existing one or a new one. The gdb was locking itself when I was in an mxd that was accessing the feature classes inside it.

No other application was accessing it at the time i.e. ArcCatalog was closed. I would reboot the VM, open a blank map document, drag in the feature class I wanted, then try to edit that feature class, or create a new feature class in that gdb. No such luck.

I mentioned that this happened for about a week. After that, everything was fine again. I’m not sure why. Windows Updates have broken file gdbs before, but I don’t think it was that.

I think that IT changed some group policies and that affected my host where the data is stored, which in turn affected the VM. I know that sounds crazy, but I’m used to accepting crazy by now when it comes to Arc. It’s still not enough to make me go to QGIS though.