Categorising spatially enabled dashboards

Lately I’ve been working with dashboards quite a bit. For clients who don’t really understand spatial data, it’s easier for them to digest the information on the map when it’s presented along with the graphs and indicators they are familiar with from Excel or Power BI.

Over the last few weeks, Julian has spent quite a bit of time setting up a number of dashboards using Operations Dashboard, each with a different purpose. On one project, we have a dashboard showing the client the real-time progress of fieldworkers on a map, along with some graphs showing the breakdown of assignments which are in progress and completed per district. It enables the client to answer questions such as which worker is causing a bottleneck. This dashboard consumes the workers and assignments layer from the Workforce for ArcGIS project, along with the various Survey123 feature services.

On another project, we have a dashboard showing the results of an asset life cycle cost analysis model. This dashboard includes graphs depicting when the client can expect to incur the greatest cost to replace key assets, as well as helping to answer questions such as: Is it cheaper to replace an asset in 5 years, or to spend an additional amount on maintenance in 3 years in order to extend the remaining useful life of the asset by 7 years?

We also have a number of ideas in dev at the moment, including the actual software we use to display the dashboard (that will have to be a post by itself). I’ve been mulling over how to package these different dashboard types as solutions to offer to a client. I decided to adapt the traditional categories to our purpose.

• Operational: This is the basic dashboard, as detailed in my first example. This type will normally display two maps – one showing real-time progress of fieldworkers and their assignments, and another showing the surveys they submit along with actual data. Graphs may include the amount of assignments completed per worker, per area or along whichever dimension is most logical (or whatever the client prefers). Filters are included to drill down through the live data.
• Analytical: This dashboard shows the results of analysing the data displayed on an operational dashboard (my second example). A single map can be used to display the analysis results per survey or per area. Graphs will vary according to client needs, but will be based on the survey points in the map. The user can interact with the dashboard by drawing various reports that they need, creating pivot tables, filtering etc.

My current dev efforts are focussed on a third type of dashboard. For a large project last year, I designed and implemented a mobile data capture solution which incorporated a QA process (to be carried out by professional engineers) as well as an invoicing process (to reduce turnaround time between carrying out the work and getting paid by the client). I’ll have to use another post to brainstorm that idea.

Survey123 support added to the ArcGIS API for Python

I don’t get excited for much these days, which is why I found my reaction to point 3 of the release notes for the 1.5.1 update so odd:

• Added support for Survey123 to the apps module

Maybe it’s because I’ve spent the last 8 months in a constant state of frustration, integrating and automating processes using Survey123, Workforce and AGOL through the API. It could be relief that I’m feeling as well. Perhaps there is light at the end of this tunnel.

Changing the worker basemap in Workforce

I’ve recently implemented Workforce for ArcGIS on a big project. It’s great being able to automatically assign sites to fieldworkers, and update their to-do lists on the fly.

However, the lack of reference data for the fieldworkers was bothering me. In the Workforce mobile app, the map displays the default topographic basemap and the location and status of their assignments. We were sending fieldworkers into areas where they would have to be a bit more vigilant of their surroundings, as well as ensure that fieldworkers did not drive through areas which were deemed as “High risk” due to crime or environmental conditions.

I modified the worker basemap of the project to include a polygon layer containing these “High risk” areas, so workers could always be aware when they were near one of these areas. I also tried changing the basemap to OpenStreetMap, but the Workforce app did not like that at all. The app crashed multiple times before I figured out it only wanted to render the default map. The dispatcher map had no issue with changing the basemap.

I also managed to overlay our own reference road network over the basemap, so the fieldworkers were aware of which direction to face when capturing a survey. This was included so that required photos were captured consistently.

Overcoming the Make Query Table bug in ArcGIS

According to my notes, I first used the Make Query Table tool in my first week at Aurecon, back in March 2012. It was the first of many, many times, because often when receiving spatial data in a non-spatial format from a non-GIS user, the first thing that gets thrown out is any trace of the original spatial component.

At some point, I realised the tool’s expression parameter was a bit wonky. As I have come up against this problem every few months since (forgetting it happens each time because I only thought to write down a note about it now), I have decided to immortalise it in a gist below.

http://desktop.arcgis.com/en/arcmap/10.3/tools/data-management-toolbox/make-query-table.htm

When inputting the optional SQL clause, ArcGIS automatically adds quotation marks “” to the field names in the dialog box. This will
pass the tool’s error checking successfully but will cause the tool to fail with an error.

If you verify the SQL clause in the dialog box, it will give a SQL error with no specifics. When adding the clause, remember to remove
the quotation marks.

e.g. If you want to join Layer1 to Layer 2 on common field ID and where Layer 1 contains “Cape Town”, ArcGIS will format your expression
in the following way:

"Layer1.ID" = "Layer2.ID" AND "Layer1.TOWN" = 'Cape Town'

You need to change it to

Layer1.ID = Layer2.ID AND Layer1.TOWN = 'Cape Town'

view raw

makequerytablebug.md

hosted with ❤ by GitHub

Create points without XY from a table in ArcPy

I had a point feature class containing features with unique names. I also had a table containing hundreds of records, each “linking” to the point feature classes via name. Normally, I could create points for the records by using Make Query Table.

I say “linking” because from looking at the data, I could see which records belonged to which point, but unfortunately there were spelling mistakes and different naming conventions e.g. if the point was called “ABC Sewage Treatment Works”, some of its matching records in the table would be “ABC WWTP”, “AB-C Waste Water Treatment Plant”.

	'''
	@author Cindy Jayakumar
	@date 31/01/2017
	– Inserts a new point with the selected geometry in to_lyr
	– Adds attributes to that point from from_lyr
	– Updates the record in from_lyr
	Uses selected features
	For the python window in ArcMap
	'''
	import arcpy
	from_lyr = r'C:\Some\Arb\Folder\work.gdb\tbl_test'
	mxd = arcpy.mapping.MapDocument("CURRENT")
	out_lyr = arcpy.mapping.ListLayers(mxd, "lyr")[0]
	'''
	– Select the single point in to_lyr that will have its geometry duplicated
	– Select the rows in from_lyr that you want to be inserted into to_lyr
	'''
	def duplicateAssets(to_lyr):
	# Matching point is selected in to_lyr
	geom = [geo[0] for geo in arcpy.da.SearchCursor(to_lyr, "SHAPE@")][0]
	# Create insert cursor on same layer
	inscursor = arcpy.da.InsertCursor(to_lyr, ("SHAPE@", "Unique_ID"))
	with arcpy.da.UpdateCursor(from_lyr, ("Unique_ID", "Matched")) as cursor:
	for row in cursor:
	# Build the new point to be inserted
	point = [geom, row[0]]
	inscursor.insertRow(point)
	print("Inserted " +str(int(row[0])))
	# Update the record with the name of the layer the point was inserted into
	row[1] = to_lyr.datasetName
	cursor.updateRow(row)
	del inscursor
	duplicatePoint(out_lyr)

view raw

duplicatePoint.py

hosted with ❤ by GitHub

Using a query table to represent a 1:M relationship spatially

I first discovered (and used) the Make Query Table tool during my second week at Aurecon (March 13 2012, according to my OneNote). This was about a month before I started using ModelBuilder (to combat the frustrations of ArcMap), and about 6 months before I started Python (to combat the frustrations of ModelBuilder).

I’m just giving a bit of context because this was before everything clicked into place for me. Before this point, I treated everything I learnt in my studies and during my internship as separate silos of information: GIS, Databases, Programming.

I did not even realise until after I was working at Aurecon that the query expressions used in ArcMap are SQL, despite me studying all those things. It just shows how one’s mindset can block progress, and how I allowed the awful experience of learning Computer Science in Afrikaans to stop me from letting things “click” for so long.

Back to the tool. Joins in ArcGIS are notoriously slow, and 1:M joins are not allowed (technically they are, but in the sense that an arbitrary matching feature will be joined). Naturally, the relationship between the GIS and the asset register is 1:M.

For example, a single point is used to represent the physical, spatial location of an asset, reservoir WRV-00001. In the asset register, this reservoir is unbundled into various components – storage tank, building, fence etc. Each of these assets have their own unique asset ID, but all have the same GIS ID.

I now need to represent all these assets spatially. The points/lines will all be on top of each other, but that’s fine. The Make Query Table tool does exactly this, but it is…quirky. I’ve compiled a list of things to remember when using this tool (supplemented by this question on GIS.SE):

1. Tables/feature classes in the relationship should be stored within the same database: I tend to remember this step only after I add the inputs and the tool shouts at me.
2. Add the feature class first in the multivalue parameter: The format of the input relies on the format of the first argument in the multivalue parameter control. The feature class should be added first to ensure that the output is a layer, otherwise it will be a table view.
3. Enclose table names and field names in quotes: For example, I wish to join the asset register table ar to the point layer points using their common field GIS_ID. By default, the tool encloses my whole expression in quotes "points.GIS_ID" = "ar.GIS_ID". This will cause the tool to fail. Add extra quotes around the field names "points"."GIS_ID" = "ar"."GIS_ID".
4. Choose the NO_KEY_FIELD option: Trying to add key fields causes some erratic behaviour (???). Just don’t do it. By selecting this option, the existing ObjectID field from the input will be used.
5. The output layer will appear to have no symbology: Go into the Layer properties, click the Symbology tab, click the existing symbol, then OK, OK. It’s a bug./li>
6. Persist to disk!: Remember to export the layer to a feature class, otherwise the layer will only exist in the map document.

Survey123: I have not been this excited in a while

Last week, I was scrolling through my Feedly and came across this post on the ESRI Australia Technical blog, one of my favourite ESRI blogs. While I am already worked up about ArcGIS Earth, it was the other app in the post that changed the course of my day (and my life).

Survey123 is an app from ESRI Labs which solves all of the issues I’ve had with mobile data capture solutions over the years. For me, it’s basically a combination of Mobenzi, GeoForms, Collector and ArcGIS Mobile.

I immediately downloaded it and did a test survey using XLSForms – another revelation to me. I can’t really say more at this stage, without testing it further, but I’ve joined the beta group on GeoNet and have already devoured the blog posts. Hopefully within the next few weeks, I will have some time to replicate one of our Mobenzi surveys in Survey123 and do a proper comparison.

CLD202x SharePoint Basics for IT Professionals: Yes please

I was planning on posting this a few weeks ago when I signed up for this course, but I’ve only gotten around to it now. I am very pleased with the course offering by Microsoft on edX, and this course is no exception.

Over 2 years ago now, when our GIS team split into Enterprise and Desktop, I was introduced to SharePoint. I had left the City just before they made switch to using SharePoint libraries as storage space instead of My Documents, so I had only dabbled with it briefly (and from an end user perspective).

During the last 2 years, I have had to interact with it in multiple ways – setting up subsites, managing permissions, creating content (such as embedding YouTube playlists, storing geometry in lists, configuring spatial viewers), and generally whatever else came up.

I never received any formal training in it, and simply clicking around in SharePoint while trying to figure it out, as I am inclined to do with other technologies, does not really work. There’s a reason this site exists. This is why I am pleased that Microsoft thought to offer this MOOC – it’s clearly needed.

The notes are very detailed, and explained very nicely in text form. This makes it very easy for me to copy into OneNote, and highlight as needed. The concepts are explained in a way that I would never have gotten from our seasoned SharePoint dev. Not that he doesn’t want to explain – he has shown me so many things on SharePoint – but it’s often the little things that one does automatically that trips up newbies.

For example, one of our clients had recently started using ArcGIS full time after only using it for very basic data capture before. He asked me why ArcMap was complaining about the projection of two feature classes being different when they were both in the same projection. I pointed out that while they were both in LO19, the one projection’s name was “LO19”, while the other one was “Transverse_Mercator_19”. It shouldn’t make a difference, but it does. The central meridian might differ by 0.00001 or something.

Another example is when one runs the Append tool in an ArcMap session, with the target table opened in the map. The tool will complete successfully, but the newly added rows won’t appear in the table even if you click away to a different table. You have to close the table, then open it again to force a refresh. Most of my ArcMap woes are solved this way – closing off whatever I’m doing and trying it again, or forcibly ending the programme. This is something I know after dealing with ArcMap for over 8 years now. It’s the GIS equivalent of

ArcGIS has a seemingly endless amount of this type of issue, and if I were teaching a total n00b how to use it, I would never even think to mention these things, because dealing with them is a reflex now. From what I can tell, SharePoint is similar, except that it probably has even more of it. Nonetheless, I’m just happy that someone has taken the time to lay out the basics in this format.

Testing out the new Collector app on Windows 10

I was very happy to see a post from ESRI appear in my feed this week about the new beta for Collector on Windows 10. It has irritated me that the only app available for Windows Phone was the ArcGIS app, which was pretty much a Collector app designed according to Modern principles anyway.

I am very pleased with this development, and am even more pleased that the app did not crash after I loaded it up on Astro. It only crashed while my colleague was attempting to navigate out of the Measure screen, but that could also just be my tablet acting up.

I will continue testing the app, hoping that related tables will appear soon, and that ESRI will turn into a universal app.

List the duplicate values in a field in an attribute table

I received some asset management data, and the state of it led me to write this script to determine if there were any values duplicated in the GIS ID field. There’s a number of ways that one can do this, but I do like to know as many methods as possible before settling on the optimised one.

	#
	# @date 29/05/2015
	# @author Cindy Williams
	#
	# Checks an attribute field for duplicate values,
	# and displays them along with the amount of duplicates.
	#
	# For use in the Python window in ArcMap.
	#
	import arcpy
	mxd = arcpy.mapping.MapDocument("CURRENT")
	lyrs = arcpy.mapping.ListLayers(mxd)
	def checkDuplicate(ftr, field):
	vals = [row[0] for row in arcpy.da.SearchCursor(ftr, field)]
	dct = {x:vals.count(x) for x in vals}
	print ftr.name
	for k, v in dct.iteritems():
	if v > 1:
	print k, v
	for lyr in lyrs:
	field_name = arcpy.ListFields(lyr, "*_ID")[0].name
	checkDuplicate(lyr, field_name)

view raw

displayDuplicateValuesInField.py

hosted with ❤ by GitHub

In Line 18, the attribute value is stored as a key in the dictionary, along with the number of occurrences of that value. In Line 21, only the keys for which there is more than one occurrence is displayed. I wrote it as a function so that I could generate a report for all the layers in the map document.

Writing this report out to a table may be more useful, especially if there are lots of duplicates, but then one may as well run Summary Statistics.