A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant

Advertisements

The end is nigh…

I read a blog post a few weeks ago about the inevitable demise of ArcMap. When ArcGIS Pro launched a couple of years ago, I immediately started preparing for the end of ArcMap. By that, I mean I played around with Pro for a few weeks then put it away until the corporate overlords decided it was time to switch.

I had to sign up for a free trial the other day for something, and found this:

What happened to ArcMap?

After logging in and heading to the downloads, I found this:

What happened to ArcMap?!?!?!!

The forums pointed it out as well. I’ve been keeping a side eye on ArcGIS Pro development over the last few years, so I’m starting the transition from October, with the aim of using it as my daily driver by December. I’ve just started training our junior consultant who comes from a CAD background on GIS as well, so I may just start him out on ArcGIS Pro from the jump.

Overcoming the Make Query Table bug in ArcGIS

According to my notes, I first used the Make Query Table tool in my first week at Aurecon, back in March 2012. It was the first of many, many times, because often when receiving spatial data in a non-spatial format from a non-GIS user, the first thing that gets thrown out is any trace of the original spatial component.

At some point, I realised the tool’s expression parameter was a bit wonky. As I have come up against this problem every few months since (forgetting it happens each time because I only thought to write down a note about it now), I have decided to immortalise it in a gist below.

Using a query table to represent a 1:M relationship spatially

I first discovered (and used) the Make Query Table tool during my second week at Aurecon (March 13 2012, according to my OneNote). This was about a month before I started using ModelBuilder (to combat the frustrations of ArcMap), and about 6 months before I started Python (to combat the frustrations of ModelBuilder).

I’m just giving a bit of context because this was before everything clicked into place for me. Before this point, I treated everything I learnt in my studies and during my internship as separate silos of information: GIS, Databases, Programming.

I did not even realise until after I was working at Aurecon that the query expressions used in ArcMap are SQL, despite me studying all those things. It just shows how one’s mindset can block progress, and how I allowed the awful experience of learning Computer Science in Afrikaans to stop me from letting things “click” for so long.

Back to the tool. Joins in ArcGIS are notoriously slow, and 1:M joins are not allowed (technically they are, but in the sense that an arbitrary matching feature will be joined). Naturally, the relationship between the GIS and the asset register is 1:M.

For example, a single point is used to represent the physical, spatial location of an asset, reservoir WRV-00001. In the asset register, this reservoir is unbundled into various components – storage tank, building, fence etc. Each of these assets have their own unique asset ID, but all have the same GIS ID.

I now need to represent all these assets spatially. The points/lines will all be on top of each other, but that’s fine. The Make Query Table tool does exactly this, but it is…quirky. I’ve compiled a list of things to remember when using this tool (supplemented by this question on GIS.SE):

  1. Tables/feature classes in the relationship should be stored within the same database: I tend to remember this step only after I add the inputs and the tool shouts at me.
  2. Add the feature class first in the multivalue parameter: The format of the input relies on the format of the first argument in the multivalue parameter control. The feature class should be added first to ensure that the output is a layer, otherwise it will be a table view.
  3. Enclose table names and field names in quotes: For example, I wish to join the asset register table ar to the point layer points using their common field GIS_ID. By default, the tool encloses my whole expression in quotes "points.GIS_ID" = "ar.GIS_ID". This will cause the tool to fail. Add extra quotes around the field names "points"."GIS_ID" = "ar"."GIS_ID".
  4. Choose the NO_KEY_FIELD option: Trying to add key fields causes some erratic behaviour (???). Just don’t do it. By selecting this option, the existing ObjectID field from the input will be used.
  5. The output layer will appear to have no symbology: Go into the Layer properties, click the Symbology tab, click the existing symbol, then OK, OK. It’s a bug./li>
  6. Persist to disk!: Remember to export the layer to a feature class, otherwise the layer will only exist in the map document.

Accessing metadata using ArcPy, Python, and now hermes!

Two weeks ago, a colleague asked me to write a script to extract some metadata values from dozens of feature classes in a gdb, and write it out to a spreadsheet along with some other descriptive properties. I fiddled around with the metadata using Python’s xml package, and managed to come up with a script for her.

So it was to my immense delight that this post popped up in my feedly on Friday. I immediately starred hermes on GitHub, and maybe this will be the first project I can actually contribute to, and not only because of the Futurama reference.

Developing an asset management GIS data maintenance methodology: Part 5 – My preparation for the way forward

(This is Part 5 of a week long series of posts on the current project I am working on, to develop a reasonable GIS data maintenance strategy for Asset Management data. Read Part 1, 2, 3, 4.)

The work I’ve done over the last few weeks (and years) is all leading to one point. A single system, with all the data topologically correct, standardised and easily accessible.

An enterprise geodatabase, with versioning enabled for the Desktop team to maintain the data without fear of conflicts. Using SDE in this manner will automatically allow for checks to be in place, for example, where I could first check the reconciled versions before moving it to default.

Archiving would be set up, and the database would be regularly backed up. Map services would be published and be made available to the non-GIS users such as the Asset team. I’d prepare the services for consumption in their app of choice: ESRI Maps for SharePoint, ArcGIS for AutoCAD, a JavaScript viewer, an Excel document with the attribute tables embedded…

The services would have the sync capability enabled, so that when they go out in the field, a Collector map could be easily configured for data capture in an offline environment. Since they visit areas which are routinely out of cell coverage, this would be ideal (and better than carrying around a printed mapbook).

While I am busy with this year’s updates, I am keeping all of this in mind. Every little bit I can do now is a little bit less that I have to do later. Once this foundation is in place, we can start looking at more advanced aspects, such as turning all this data into geometric networks, and creating custom tools which can automatically calculate the remaining useful life, asset depreciation and answer all the questions that could possibly be asked.

Developing an asset management GIS data maintenance methodology: Part 4 – Data handover and the audit process

(This is Part 4 of a week long series of posts on the current project I am working on, to develop a reasonable GIS data maintenance strategy for Asset Management data. Read Part 1, Part 2 and Part 3 here).

Once all the new data has been processed and captured, or existing data flagged as asset upgrades, we have to extract several attributes for use by the asset guys. This includes the all important GIS ID, which will be used to link the asset register back to the GIS. The asset register contains a mountain of information about the status of the assets, remaining useful life and other financial information. In the GIS data, we are purely concerned with geometry: is this asset in its correct physical location, with the correct dimensions?

We will carry a few other attributes, such as material type, diameter of pipe, name (if relevant) and town. Once I’ve compiled all the data into this structure, I convert it to a spreadsheet filtered by feature type and including the lengths of the line data. The GIS length is used over the length supplied by the client as it represents the situation on the ground.

Once everything has been submitted to the client, a short while later the auditors will appear. Using some algorithm, they select a sample of the assets and send us a list of GIS IDs or asset names. I link this list up to our GIS and provide a KML file of the locations. This proves that we know where those assets listed on the asset register are (and that no one is trying to fabricate assets). Sometimes the auditors will only provide the asset names – that makes it a bit harder for me to link since we don’t always have the name attribute. It’s the reason why the GIS ID should appear in all documentation.

For one client, I went along with the project leader and sat with an auditor in front of ArcMap. I had to physically look up the assets as she read out the GIS ID or asset name. At that time, the data was not in a good state to begin with, so fortunately she had only sampled assets that I could easily retrieve. As part of this methodology development, I’m documenting the state of the data as it was received, the problems it causes (including unnecessary delays) and the estimated time it would take to clean up the full database for each client.

That’s the part I’m most excited about in this process. I can finally get my hands into the data and standardise it, so if there are any auditor queries, or someone wants to know something quickly about a particular asset, I will know exactly where to find it.