A productive week or so!

The last couple of weeks have been very productive for the EngLaId team in many respects.

We are extremely grateful to Nick Boldrini, Lesley Mitchell, and Bryn Tapper from Durham, Greater Manchester and Cornwall HERs respectively, for extracting HER data for their areas for the EngLaId project. Bryn and Nick have been really helpful both in terms of providing detailed advice about the HER data from their areas and offering useful suggestions with regards to some of the project’s future outputs. We are particularly grateful to Lesley for managing to extract the data for the project prior to the temporary closure of the Greater Manchester HER while it relocates to Salford.

EngLaId’s recent progress with populating the project database has been assisted greatly by the commissioning, at the suggestion of Mike Hemblade (North Lincolnshire HER) and Penny Ward (Shropshire HER), of an ExeGesIS query to extract bulk downloads of HER data from HERs using HBSMR software (http://www.esdm.co.uk/hbsmr.asp). This includes the vast majority of HERs in England and more broadly. The query will hopefully facilitate greatly for HER Officers, the process of extracting the significant volume of data that the EngLaId project requires. We really appreciate the ExeGesIS team’s promptness in producing this query.

More broadly, Chris Green (GIS) and John Pybus (eResources) had some good feedback on the papers they gave at the Computer Applications in Archaeology conference in Southampton (more to follow). Letty Ten Harkel (early medieval researcher) is progressing well with her paper for the forthcoming international Landscape Archaeology Conference in Berlin (http://www.geo.fu-berlin.de/geog/fachrichtungen/physgeog/lac2012/). Meanwhile Miranda Cresswell (project artist) has been making the most of the sunshine and working intensively on drawings at both Port Meadow in Oxford and at Danebury Iron Age hillfort (see http://visualenglaid.wordpress.com/ for the latest updates).

Finally, Laura Morley (research coordinator) thoroughly enjoyed her visit to Preston to the NW regional HER meeting on 28th March. As has been the case at all of the regional HER meetings the EngLaId team has participated in recently, the discussions held were very constructive. Helpful suggestions and advice from those involved regarding the project’s research plans were gratefully received.

DPhil studentships

This is just a quick post to advertise the fact that the deadlines have been extended for the three DPhil studentships associated with the EngLaId project.  Details can be found here.  These studentships are now open for applications until the end of June 2012, so please apply if you think you’re the right person for one of the positions and would like to be a part of our exciting project!

Meeting with EH; CAA 2012

The EngLaId team had a productive meeting with Simon Crutchley and Martin Newman from English Heritage last week, where we presented our work to date on their data (the NMP and its associated attribute database AMIE/NRHE), and discussed how we might combine / synthesise their data with our other datasets.  We also discussed many other topics, including the history of the NMP.  Our thanks go to Simon and Martin for coming up to Oxford to meet the team.

Meeting with EH
Meeting with EH

On another note, John Pybus* and Chris Green** will be presenting next week at the Computer Applications in Archaeology conference in Southampton.  If you’re going, see you there!

* Tuesday March 27, 2012 11:15am – 4:00pm @ Building 65, 1097 Streamed into room 1163

** Thursday March 29, 2012 9:00am – 1:15pm @ Building 65, Lecture Theatre A

Regional HER Meetings

The EngLaId team would like to thank all those present at the regional HER meetings for Yorkshire, the North East and Humber (7 March), the South East (8 March) and the East Midlands (15 March) for their feedback and stimulating discussion!

HER Regional Meeting 7 March 2012
HER Regional Meeting 7 March 2012

ArcMap: generating squares to represent NGR precision

Often when dealing with spatial data recorded using Ordnance Survey (OS) National Grid References (NGRs for short), you will find that different objects within your data are recorded to differing levels of spatial precision.  This can lead to misrepresentative results if treated simply as point objects (especially for objects poessessing only a very vague location) and, therefore, we need a method of representation that is truer to the actual spatial precision of the data objects.

To help explain, NGRs are formatted as follows (fee free to skip this paragraph if you understand NGRs): the OS divides the UK up into 100km by 100km grid squares, which are designated using two letters.  Therefore, an NGR consists of two letters followed by a string of numbers, the numbers representing the object’s location within the large grid square.  There should be an even number of these digits, with the first half representing the easting and the second half the northing.  However, the exact number of digits will vary according to how precisely located the object is.  For example, an NGR such as SQ2222233333 is in the square SQ at the location 22222 metres east, 33333 metres north; however we would have to add zeroes to a shorter NGR as the precision of the point is less well known, e.g. SP444555 is in the square SP at the location 44400, 55500.  For more on converting NGRs to numeric coordinates, see my previous post.

However, simply counting the digits is not always a sufficient measure of coordinate precision, as the person who entered the data may have tagged zeroes onto the end of the easting and northing.  For example, the NGR ST5500066000 appears to be accurate to the nearest metre, but it is unlikely that a point would fall exactly onto the 55000, 66000 point and, as such, it is much more likely that this point is recorded to the nearest kilometre.  I have a small Python script to perform this test, which can be downloaded here.

In any event, the end result of this is that an NGR that is apparently a point is, in fact, properly the origin point (southwest corner) of a square of width and height equal to its precision, e.g. the NGR ST5566 represents a square 1km by 1km spreading northeast from its location, whereas the NGR SQ2222233333 is a 1m by 1m square.  Therefore, if we want to represent these points properly in a GIS environment, we have to present them as squares of appropriate size.

So how can we automate creating this representation in ArcMap (version 10)?*  I will assume reasonable familiarity with the software.

If we have a point shapefile of our locations (use a duplicate to preserve the original), with their associated attribute data including their spatial precision**, we can create these squares in four relatively simple steps.  Here are our points on a map:

The points...

First, we open up the attribute table for the layer.  Then we add a new field to the table, which we shall call MIDPOINT and which will be of the Float type (i.e. a floating point number).  Then we open up the Field Calculator for the MIDPOINT field and enter the calculation [Precision field] / 2.0 as follows:

Field Calculator (i)
Calculating the midpoint of our square.

This new field will be used to add on to the x and y coordinates of each point in the layer to move them to the centre of each square to be created.  We then open the Field Calculator for the Shape field and enter the code snippet seen here (make sure you have the Python parser selected at the top):

Field Calculator (II)
Moving the points.

The expression can be downloaded here.  This results in the points moving to a new location to the north east in the midpoint of the squares we will soon create (notice that some points have moved a long way yet others remain close to their origin):

Moved points
New point locations plotted in red against old in black.

Next we create buffers for our moved points using the Buffer tool in ArcToolbox.  You want to set it to use the MIDPOINT field as the buffer size (we use the MIDPOINT rather than the full precision field as the size will be the radius of the circle created, not the diameter) and you do not want it to dissolve the buffers where they overlap, as we want one object for each point.  The result should look something like this:

Buffers of our points.

Finally, we use the Feature Envelope to Polygon tool in ArcToolbox using the buffer layer as input.  This, helpfully, retrieves the envelopes that enclose each circle (used to make spatial queries faster etc.), which is equivalent to the squares that we were after in the first place.  Therefore, this layer is the result that we desired:

Final result
The final result of the process.

Any points that do not seem to have output squares in the image do, in fact, have them but they are too small to see at this scale (e.g. 1m or 10m precision).  Overall, this is a simple way to do this task in just a few minutes whilst avoiding any overly complex scripting.  I think I will work on a Python script to achieve this task in one step, but the method just described works perfectly well in the interim.

The final result means that we now both have a visual representation of the area to which each of our NGR “points” actually relates, and also a layer which can be used in further analyses when trying to understand the contents of our layer: all of the layer attributes from the original layer should find their way automatically into the final output, so we can perform whatever analyses we wish on our new, more properly representative geographic objects.

Chris Green

* ArcMap is part of ESRI’s ArcGIS and is the GIS software that we are predominantly using in this project.

** If you did not record the precision of the NGRs whilst converting them, you could use the Field Calculator to work it out (in the same way as described above for the other calculations).  You would have to create a fields called something like ‘PRECISION’ and ‘TRUEPRECIS’, and then use the expression code downloadable here and here in turn on the first and then second fields.

The AIP: maximising the picture

The Archaeological Investigations Project (AIP) is an ongoing project at the University of Bournemouth that collates the brief results of commercial archaeological investigations in England for each year.  With the very kind assistance of their Ehren Milner, we have extracted data from their database for our period in respect of two specific classes of monument type: settlements and field systems.  These two search queries were selected first as they seemed to me to be two of the key monument types of interest to our project: we will most probably ask Ehren to undertake further, different or revised queries in addition in the future.

The search terms used were taken from the English Heritage thesaurus, as that is used by many organisations and it seemed a sensible starting point.  Terms were selected from the list where they were relevant to our period of interest (e.g. we did not include terms under the settlement class like ‘Olympic village’ or ‘railway workers temporary settlement’), leaving out some longer terms which ought to be captured by a subset of their content (e.g. by including ‘field system’, we also ought to catch items named ‘Celtic field system’, ‘coaxial field system’, ‘enclosed field system’, etc.).

The field system query term list was:


The settlement query term list was:


It is not claimed that these lists are exhaustive or likely to capture every instance of these two classes of monument.  For instance, under settlement I was uncertain whether I should have included ‘villa’ or not, especially as I did include terms like ‘enclosure’ and ‘fort’.  The searches would also not capture objects recorded using any different terminologies, or sites of too small a size to be recorded within monument types of this sort of scale (e.g. an intervention which only discovered one ditch of a field system or settlement could not properly be recorded as such by the excavator without further supportive evidence, so would likely be recorded in the AIP as simply ‘ditch’).

In any event, subject to these caveats, the data received from Ehren (in .mdb database format) was exported to a .csv table and processed using a Python script to convert the OS NGRs (see my previous post) to numeric x and y coordinates (only 3 sites of over 4,000 seemed to have incorrectly recorded NGRs, i.e. they had an odd number of digits) and to extract the periods recorded for each site to a separate field in the new output table.  This processed table can then be imported into ArcGIS in the conventional fashion.

Here is a map of the results, plotted against our initial proposed case study locations:

AIP settlements and field systems (Bronze Age to early medieval)
Settlements and field systems (Bronze Age to early medieval) taken from AIP data and plotted against proposed case study areas.

This data is especially useful as it helps to fill gaps between areas where the National Mapping Programme (NMP) has not yet been undertaken.  For example, Cambridgeshire has had much commercial archaeology done in recent years, but has not been surveyed by the NMP as yet.  Hopefully, by using data from the HERs, EH data, PAS data, and AIP data in combination (alongside other more specific datasets), we ought to be able to build up an excellent picture of English archaeology for our period.

Chris Green