A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant

Thoughts on spatial data warehousing

I’ve been thinking quite a bit lately about how to store spatial data. It’s something I’ve covered here and my attitude towards this topic has evolved over the years.

The organisations I’ve worked in have mountains of spatial data accumulated over the years. The data is stored in shapefiles, geodatabases, normal databases, spreadsheets, documents, reports, photos…Why is it this way? It doesn’t have to be this way. It shouldn’t be this way!

In the course of my research for a topic for my project for next year, I’ve honed in on the methods for implementing an enterprise geoportal within an existing spatial data infrastructure. However, I feel like my focus is shifting to the data that the geoportal is trying to expose to a larger audience.

The concepts of a spatial data warehouse and a spatially enabled operational data store have been intriguing me. A regular GIS task involves comparing spatial data across a time period, analysing trends and presenting the results in a map or report. Why aren’t we storing this historical data in a SDW that’s optimised for reporting?

Non-spatial data can come from a variety of sources as well – spreadsheets, other databases etc. Another common GIS task is to spatially enable these datasets. Why are we not storing the outputs in a spatial enabled operational data store in an open format like GML?

I think it’s because to plan and implement a SDW/S-ODS takes time (and money). With a normal EDW, the organisation will not need much convincing to see the benefit of implementing one. “Spatial” is still seen as an “add-on”, or a “nice-to-have”.

Is 2016 the year of location?

I came across an interesting post on The Next Web the other day. It asks if 2016 is the year of location. The premise of this blog is all about that – the fact that everything we do is tied to a location. Everything is spatial.

I have believed for quite a while now that as GIS grows more and more, and becomes more integrated into every domain, that the role of the GIS professional will shift drastically. The “pure” GIS professional will be enterprise/programming focussed, supporting the domain GIS users who aren’t necessarily trained GIS users, but merely use GIS as part of their toolset.

ArcMap Woes Part 6: Random failure of gp tools on grouped layers

I had a series of errors recently while I was trying to run a few geoprocessing tools manually on layers within a group layer. The error message kept saying it couldn’t open the layer, even though the layer was clearly in the map. Even restarting ArcMap did not help.

I eventually discovered that geoprocessing tools may randomly fail on layers which are part of a group layer. I say randomly because sometimes the tools would run as expected, other times not. I’ve only experienced this anomaly when using the ArcMap interface – running the same sequence of tools on all the layers in a group layer in a for loop in Python does not give this error.

Geodesign: Change Your World MOOC: I think I’ll keep my world the same for now.

I was so excited to see that Penn State was offering another MOOC this year, this time about geodesign. It’s been on my watchlist for the past few months, so when I got the email to announce that class was starting, I immediately checked it out.

After my last Penn State MOOC though, I was a bit more wary this time around. A cursory glance down the left pane showed me what I was hoping I would not find:

I half-heartedly went through the course content after that, but I had already decided not to carry on. The peer assessment assignment is 40% of the grade for the course. I’ve mentioned here before that peer assessments in a MOOC are a dealbreaker for me. I learn best on my own. I enjoy working as part of a team, but on my own as part of a team.

This week I started with the two other modules I need to get into Hons. The one is about software engineering principles, while the other is about advanced database development using an old version of Oracle (why, Unisa, why). I’m quite excited about the SQL one, as it will give me more opportunities to flex my SQL muscles, and to get exposure to a different database platform (something other than SQL Server).

Basically, even if I wanted to do this MOOC now, I can’t. Best of luck to the Geodesign students, and hopefully with the next offering, they will relax the peer assessment requirement.

ArcMap Woes Part 5: Exporting map without options

Sometimes, when in ArcMap, and one wants to export a map as a JPG or PNG or something, there won’t be any options. You can’t change the resolution, can’t change the export quality, can’t include a world file or anything. This used to frustrate me enormously, because then I would have to restart ArcMap to get the options to appear.

Sometimes even that wouldn’t work. I would then have to disturb a colleague and have them open the same mxd, to find that the options have miraculously reappeared. I have since figured out that if this happens, one needs to cancel the export dialog, switch to layout view and then export the map.

I am aware that it is possible to export the map with options while in data view; it’s just a random thing that happens. At least I know how to solve this particular issue.

ArcMap Woes Part 4: Permissions errors

Sometimes when I’m in ArcMap, I can’t start an edit session. Normally, I restart ArcMap and everything is fine.

Recently, for about a week, I could not edit inside any file gdb, or create new feature classes inside of an existing one or a new one. The gdb was locking itself when I was in an mxd that was accessing the feature classes inside it.

No other application was accessing it at the time i.e. ArcCatalog was closed. I would reboot the VM, open a blank map document, drag in the feature class I wanted, then try to edit that feature class, or create a new feature class in that gdb. No such luck.

I mentioned that this happened for about a week. After that, everything was fine again. I’m not sure why. Windows Updates have broken file gdbs before, but I don’t think it was that.

I think that IT changed some group policies and that affected my host where the data is stored, which in turn affected the VM. I know that sounds crazy, but I’m used to accepting crazy by now when it comes to Arc. It’s still not enough to make me go to QGIS though.

ArcMap Woes Part 3: Mxds that get tired

I tend to refresh my map documents every 3 days or so. By that, I mean I drag all my layers into a blank mxd, and save that as my new working document.

After a round of analysis and editing (the length of the round varies), the current mxd seems to get “tired”. ArcMap takes a few seconds longer to respond if I make a change, open a tool or do anything. Closing that mxd and opening a different one results in ArcMap functioning as expected; try opening that first mxd again, and it’s back to slowness and frustration.

This tends to happen particularly with mxds containing CAD drawings, or ones I’ve used to perform a lot of edits in. I’m not sure why it happens, all I know is that if ArcMap starts “getting slow”, I open a blank mxd and carry on. I don’t fight it, I just accept it for what it is.

Now that ArcGIS Pro is doing rounds, I’m hoping that this won’t be an issue anymore.

This doesn’t bother me as much as it did before, as I do most of my work/processing through Python now anyway. If I really need to, I open ArcCatalog to look at something.

The problem gets a bit more serious when geoprocessing tools start failing randomly, and you spend three hours checking your inputs, modifying parameters and pulling your hair out before you remember to open a blank document, run the same tool in there and get the results you were expecting.

Geodeveloper – A declaration.

For a while now, there has seemed to be an increase in the use of the prefix geo. I remember being a bit annoyed about it – I’d be reading some random news article and the author would talk about “geoblocking” or “geofencing” or – help us – “geolocation”. I realised my annoyance stemmed from the fact that the terms were often used in the wrong context, or that the author seemed to misunderstand their definitions.

That’s different now. Over the past few weeks, I’ve been learning how to use ESRI’s GeoEvent Extension for ArcGIS for Server (could that name be any longer?). I’m learning how to set up a geofence to geoblock end users based on their geolocation (ha!).

While I will elaborate more on my adventures with GeoEvent in another post, a couple of other things have happened over the past two months which have led me to rethink my stance on what I do. The nature of my work as a GIS Technician on the Enterprise GIS team requires me to be good at a number of different things, including things which aren’t strictly GIS. That’s fine – I love doing it, and I enjoy determining what the best combination of technologies would be to solve a particular problem.

However, there is one particular aspect of my work which is a continuous challenge, which constantly forces me to adapt, learn, research, and improve my skills in it everyday, because there is always something new. Programming. Specifically, programming for GIS, although lately I have been stepping outside of that arena as part of a side project.

It’s where I’m focussing my efforts whenever I have “free” time. All the GIS work I do, my studies at UNISA in preparation for Honours, my MOOCs – it’s all for one thing: to be a geodeveloper.

Geodeveloper (n): A developer who specialises in, and focuses on, programming specifically for use within the GIS/Geoinformatics field.

DEV203x Introduction to Bootstrap: Uh-oh

I was very excited to see that Microsoft is now offering MOOCs through edX. I immediately signed up for Introduction to Bootstrap and Transact-SQL. Bootstrap opened up on 1 April, so I spent some time getting the videos, downloading the notes and preparing my OneNote section for it.

I idly clicked over to the first lab, which looked promising – a detailed assignment, clearly designed to test the knowledge gained from the videos and notes, with clear submission rules. I then scrolled to the bottom and saw this:

I’ve mentioned in several previous posts that peer assessments in a MOOC is a dealbreaker for me. I’ve changed my opinion somewhat on the use of video lectures (just barely though), but making peer assessments a requirement is a bit much for me.

While my need to finish what I’ve started is strong, I have many other things to do as well, so for this one I may be content to just grab the content for self-paced learning/reference at a later stage.