Categorising spatially enabled dashboards

Lately I’ve been working with dashboards quite a bit. For clients who don’t really understand spatial data, it’s easier for them to digest the information on the map when it’s presented along with the graphs and indicators they are familiar with from Excel or Power BI.

Over the last few weeks, Julian has spent quite a bit of time setting up a number of dashboards using Operations Dashboard, each with a different purpose. On one project, we have a dashboard showing the client the real-time progress of fieldworkers on a map, along with some graphs showing the breakdown of assignments which are in progress and completed per district. It enables the client to answer questions such as which worker is causing a bottleneck. This dashboard consumes the workers and assignments layer from the Workforce for ArcGIS project, along with the various Survey123 feature services.

On another project, we have a dashboard showing the results of an asset life cycle cost analysis model. This dashboard includes graphs depicting when the client can expect to incur the greatest cost to replace key assets, as well as helping to answer questions such as: Is it cheaper to replace an asset in 5 years, or to spend an additional amount on maintenance in 3 years in order to extend the remaining useful life of the asset by 7 years?

We also have a number of ideas in dev at the moment, including the actual software we use to display the dashboard (that will have to be a post by itself). I’ve been mulling over how to package these different dashboard types as solutions to offer to a client. I decided to adapt the traditional categories to our purpose.

  • Operational: This is the basic dashboard, as detailed in my first example. This type will normally display two maps – one showing real-time progress of fieldworkers and their assignments, and another showing the surveys they submit along with actual data. Graphs may include the amount of assignments completed per worker, per area or along whichever dimension is most logical (or whatever the client prefers). Filters are included to drill down through the live data.
  • Analytical: This dashboard shows the results of analysing the data displayed on an operational dashboard (my second example). A single map can be used to display the analysis results per survey or per area. Graphs will vary according to client needs, but will be based on the survey points in the map. The user can interact with the dashboard by drawing various reports that they need, creating pivot tables, filtering etc.

My current dev efforts are focussed on a third type of dashboard. For a large project last year, I designed and implemented a mobile data capture solution which incorporated a QA process (to be carried out by professional engineers) as well as an invoicing process (to reduce turnaround time between carrying out the work and getting paid by the client). I’ll have to use another post to brainstorm that idea.

How to convince someone to move from paper-based forms to electronic surveys

I spent March of 2018 working closely with a bridge engineer. For years, they had been using four paper questionnaires to capture inventory and inspection data of structures such as bridges, culverts, gantries etc. They would arrive at a structure, select the appropriate form, measure and write everything down, take a load of photos, draw a sketch or two and move on to the next one.

When they got back to the office, they would get a few students to manually input the captured data into an electronic form, which would store it in a database on their local machine. The students would also need to manually link photos to the correct structure. As with most projects, time would always run out, so the engineers would also need to help out with this electronic transfer process.

Once the screaming in my head subsided, I asked him, “Why? Why is it being done this way? It’s 2018. The process you are describing should have been phasing out at least 5 years ago already.” He didn’t really have a straight answer for me, beyond “this is the way it’s always been done”.

Over the next few weeks, I showed him that there is a better way. I used the first week to convert the largest of the four forms using Survey123. The form was fairly complex – with no access to a table layout or even a grid theme, I had to make a number of design decisions which wouldn’t impact the user experience too much while still retaining (and even enhancing) the functionality available in the paper form.

After my initial stab at it, we spent 2 more weeks going back and forth, fine-tuning choice lists, removing unnecessary questions, changing section groups, enforcing relevant fields, choosing repeats and optimising calculations. We ended up with an xls of 250 rows of questions and 35 choice lists. I was fortunate to have access to one of the databases they’d used on a previous project, so I was able to extract the choices from the lookup tables using SQL and Python.

I spent the last week of March replicating the other three forms. They were similar enough that I could copy and paste much of what I had implemented on the first form, but different enough that I couldn’t keep everything in one form. Once that was completed, I published all the surveys and gave it to him to thoroughly test (I believe my exact words were “Try to break it”).

By the first week of April, I had fixed most of the bugs and we were ready to train the students on it in the field. I set up a web app allowing him to view the surveys as they were submitted. He could immediately send a message to the students in the WhatsApp group I set up on their tablets if they were measuring components incorrectly, or not describing items properly. He was fully converted.

In my next post, I’ll detail how I took everything down and rebuilt it from its ashes by adding Workforce for ArcGIS and Operations Dashboard to create a more efficient system.

ArcGIS Pro: A year later

I mentioned previously that I was finally considering making the jump from ArcMap to ArcGIS Pro. I made the switch in early 2018, and after an adjustment period of a few weeks, I have not looked back.

Seriously.

For those still waiting to make the switch, just do it. I highly recommend completing the official tutorials. It is a huge adjustment to go from dialogue boxes to the ribbon interface, akin to the user shock experience when switching over from Office 2003 to Office 2007.

You’ll also need to learn how to deal with multiple data  map frames and layouts in one map document project, the lack of a standalone ArcCatalog, and the ability to link 2D and 3D views in one place. There are also new terms which you will need to learn, and new concepts.

A few of my favourite things:

  • Seamless integration with AGOL/Portal
  • Labelling properties now accessible through groups on the ribbon – no more Inception levels of dialogue boxes
  • Data driven pages is more robust
  • BIM integration
  • arcpy.mapping overhauled to arcpy.mp – a much more intuitive module
  • One ArcGIS project per client containing multiple data frames for each task – no need for dozens of map documents
  • Python 3 support (and built in Anaconda!)

I could go on. Our local licence server for Desktop went down a few weeks ago, and I only found out a few days after it happened because I just don’t open ArcMap anymore. With the latest update to Pro, I can now quickly launch a project without needing to save it first, a feature that was lacking up until now. My last reason to use ArcMap is gone.

Changing the worker basemap in Workforce

I’ve recently implemented Workforce for ArcGIS on a big project. It’s great being able to automatically assign sites to fieldworkers, and update their to-do lists on the fly.

However, the lack of reference data for the fieldworkers was bothering me. In the Workforce mobile app, the map displays the default topographic basemap and the location and status of their assignments. We were sending fieldworkers into areas where they would have to be a bit more vigilant of their surroundings, as well as ensure that fieldworkers did not drive through areas which were deemed as “High risk” due to crime or environmental conditions.

I modified the worker basemap of the project to include a polygon layer containing these “High risk” areas, so workers could always be aware when they were near one of these areas. I also tried changing the basemap to OpenStreetMap, but the Workforce app did not like that at all. The app crashed multiple times before I figured out it only wanted to render the default map. The dispatcher map had no issue with changing the basemap.

I also managed to overlay our own reference road network over the basemap, so the fieldworkers were aware of which direction to face when capturing a survey. This was included so that required photos were captured consistently.

A rant about attachments

I remember when attachments were first introduced in ArcGIS Desktop (10.2 I think? whoops it was ArcGIS 10). It was a very useful feature, and more functionality was added over the years.

It also made mobile data capture even easier. The fieldworkers would go out, do their assessments, and attach multiple photos to their points. However, attachments with Collector has caused me so much frustration. Specifically, syncing with attachments.

The nature of the work we do (and the economic environment we are in) means that by default, I take the maps offline so that the fieldworkers can carry out their assessments, and then sync back to AGOL when they are on lunch break (or whenever they can pick up WiFi). I discovered a few years ago that once one hits a certain threshold (like 20 attachments in the map), there are going to be problems syncing.

It will just outright fail, or take very long and may need to be attempted a number of times. Why is this? I don’t know. Over the years, I’ve encountered this issue on all types of devices – the latest iPhones, low-end Android tablets, high-end Android tablets, mid-range Android phones…

What it seems like to me is that Collector “expects” a certain connection speed, and when it doesn’t get it, it times out and rolls back the sync. Fair enough – I’ve found multiple delta tables on devices I’ve needed to recover the databases from due to failed sync attempts. On a current project, they are using rugged devices which have really awful network chips (as in, I need to stand about 1 or 2m away from the access point so that I can take the maps offline). Naturally, at the end of the first day, each device had dozens of features with multiple attachments each, which refused to sync.

They have been out in the field for 2 weeks. Everyday, I have to manually retrieve the databases from the device, recover them, and push them out into appropriate geodatabases once I’ve determined what’s inside them.

So clear

I can deal with all of that, because Python is a tool that I maaay have mentioned here before. What I cannot deal with is the fact that attachments are still lost during geoprocessing. The fact that it was added as an environment setting in ArcGIS 10.5 and has been available in ArcGIS Pro for a while is of little comfort to me as I currently have access to neither.

Fine. I store the GlobalIDs in another field, merge the features together into their correct feature classes, enable attachments and insert the records from the corresponding attachment tables. Of course, I forget that the relationship class is now messed up, as it’s linking through the (now incorrect) GlobalID fields instead of the fields I stored the original IDs in.

After staring at the screen cross-eyed, I then realise that I only need to provide the attachments as jpgs in a folder, which I can extract from the tables using the original IDs and write into subfolders based on the feature type. I don’t actually need to link them back together since the technician does not need to view the photos to complete the work in ArcMap. /endrant

The end is nigh…

I read a blog post a few weeks ago about the inevitable demise of ArcMap. When ArcGIS Pro launched a couple of years ago, I immediately started preparing for the end of ArcMap. By that, I mean I played around with Pro for a few weeks then put it away until the corporate overlords decided it was time to switch.

I had to sign up for a free trial the other day for something, and found this:

What happened to ArcMap?

After logging in and heading to the downloads, I found this:

What happened to ArcMap?!?!?!!

The forums pointed it out as well. I’ve been keeping a side eye on ArcGIS Pro development over the last few years, so I’m starting the transition from October, with the aim of using it as my daily driver by December. I’ve just started training our junior consultant who comes from a CAD background on GIS as well, so I may just start him out on ArcGIS Pro from the jump.

Overcoming the Make Query Table bug in ArcGIS

According to my notes, I first used the Make Query Table tool in my first week at Aurecon, back in March 2012. It was the first of many, many times, because often when receiving spatial data in a non-spatial format from a non-GIS user, the first thing that gets thrown out is any trace of the original spatial component.

At some point, I realised the tool’s expression parameter was a bit wonky. As I have come up against this problem every few months since (forgetting it happens each time because I only thought to write down a note about it now), I have decided to immortalise it in a gist below.