Extracting trends (III)

Following again on from my previous two posts (1)(2), I have been experimenting further with constructing trend surfaces, this time for specific sub-sets of my downloaded AIP data for evaluations and post-determination / research results from 1990 to 2010.

First, I removed all of the data for investigations that had no results in terms of dated features, which results in a very similar trend surface to that for all of the data including investigations with no substantive positive evidence:

1 AIP_trend_noNegEvid
12th power trend surface for AIP data (excluding investigations with no positive results)

Then I constructed trend surfaces for the same data but filtered down to investigations producing results for each of EngLaId’s four main broad time periods:

2 AIP_trend_BA
12th power trend surface for AIP data (Bronze Age)
3 AIP_trend_IA
12th power trend surface for AIP data (Iron Age)
4 AIP_trend_RO
12th power trend surface for AIP data (Roman)
5 AIP_trend_EM
12th power trend surface for AIP data (early medieval)

These results all look quite interesting to me, especially as they all vary quite significantly from the overall trend for all periods (albeit this is less the case for the Roman data).  The Bronze Age data shows a very clear bias towards an arc across south-eastern England from Dorset through to Kent and up into parts of East Anglia (the dry bits essentially), with the exception of the South Downs and the Weald.  The Iron Age is very strongly biased towards the counties north of London up to Cambridgeshire, across to north-east Kent and along the south coast.  There is also more of a northern trend than in the Bronze Age, with quite a significant peak in East Yorkshire.  The Roman data is distinctly biased towards London, Kent, the south coast, East Yorkshire and the Severn estuary region.  There is a surprising lack of any significant peak in the Tyneside area, considering the significant peak there in the data for all periods and the presence of Hadrian’s Wall.  For the early medieval, there is a very clear bias towards eastern England around the Fens and towards Kent.

I particularly like these results as they largely differ so significantly from the overall trend for all periods, which suggests that these patterns are more likely to be due to genuine distributions of underlying archaeological data, not just due to patterns of modern fieldwork (albeit this will still remain a very significant factor).  I am not sure any of the results are particularly surprising, interpretively, but they do confirm for me that we can extract spatial patterning from AIP data that is not just wholly biased towards areas of significant modern development.

Chris Green

Extracting trends

One particularly major modern bias that exists in the datasets being studied by EngLaId is that of where archaeological activity tends to take place in England.  In particular, commercial archaeology tends to take place more commonly in areas where more modern development takes place (especially relevant to our AIP and HER datasets) and also there are obvious biases in the PAS towards areas which are more popular with metal detectorists /  have better conditions for metal detecting / where Finds Liaison Officers are more well-established.

It is necessary to somehow quantify these biases, so that it can be discerned whether patterns discovered are more likely to be a true reflection of past activity or more likely to be an artifact of modern archaeological activity.  Amongst others, Andrew Bevan has done some very interesting work in this area using kernel density estimates (Bevan 2012), but I have my own bias towards a slightly different approach: trend surface modelling.

The trend surface is, essentially, an attempt to model underlying trends in a point based dataset based upon numerical values attached to those points.  The algorithm creates a polynomial surface which tries to reflect that trend.  It is then possible to use that surface to test whether individual points within the original data either fall above or below the expected value (bucking the trend), or fall more close to the trend itself.  This would then require explanation.  The first step, however, is creating the trend surface.

As a measure of where commercial archaeological interventions have taken place since the onset of PPG16 in 1990, I first extracted AIP data for each county of England via their website, restricting my query to the years 1990 to 2010 and to field evaluations and post-determination / non-planning events.  These spreadsheets were converted, combined, and imported into ArcGIS using my script.  The resulting points were then counted per 1 x 1 km grid cell and a 12th order (the most complex surface available in ArcGIS) trend surface created:

12th order polynomial linear trend surface for field evaluations and post-determination / non-planning events for 1990 to 2010 recorded by the AIP.

Essentially, this map then shows the expected amount of interventions per square kilometre across England.  Although the values are small (0-2+), the pattern looks convincing to me, with obvious peaks in London, Kent, the eastern south coast, the Bristol region, South / East Yorkshire and Tyneside.  There are also clear troughs across most upland regions of England.  It is particularly obvious how much commercial archaeological work has taken place in London over the past twenty years.

I then repeated this task for the PAS data for all time periods (as of August 2012):

12th order polynomial linear trend surface for all PAS finds up to August 2012.

Again, this map shows the expected number of finds per square kilometre across England.  Here we see obvious peaks in East Anglia, Kent, the eastern South coast and the Isle of Wight, western Cornwall, Northamptonshire, and Humberside / Lincolnshire.  Again, there are obvious troughs across the uplands of England and also in some areas of dense settlement (Tyneside, Essex, Medway).  As stated above, there are several factors at play here, but the pattern seems a convincing and useful one to my eyes.

The next stage for the PAS data would be to construct further trend surfaces for each of our time periods and see how these compare against the overall trend across all periods.  In this way, it ought to be possible to pick out areas for further study which show particular peaks within a single time period that are not present in the overall trend.  Whether this works, only time will tell!

Chris Green


Bevan, A. 2012. “Spatial methods for analysing large-scale artefact inventories.”  Antiquity 86, pp. 492-506.

The Archaeological Investigations Project (AIP)

As previously mentioned, the Archaeological Investigations Project (AIP), based at the University of Bournemouth, has been gathering brief details of developer-funded archaeology that has taken place in England since 1990.  They provide an excellent, useful resource for archaeological researchers, but unfortunately we understand that their funding has now come to an end, at least until early 2013.  We were particularly sad to hear that Ehren Milner was losing his job, as he has been very helpful to us in our work on the EngLaId project.  We wish Ehren all the best for the future and hope that the AIP is able to continue their excellent work at some point soon.

In the meanwhile and until they are back up and running properly, I have written some Python code to convert the “xls” files downloadable from their website into a data format more suited to GIS analysis (i.e. a shapefile).  This is complicated slightly by the fact that the downloads are not, in fact, xls files but tables coded in html.  The code is rather EngLaId specific in some of its content, but I thought I would share it here in case it is of use to anyone else using AIP data.  The code should be available here:


It is probably a bit overcomplicated and not especially efficient (as I wrote it for my own occasional use only, primarily), but it might help anyone else trying to do similar things.  I cannot promise that it will work 100% of the time, but it seems to have worked for me!  The code is commented, so adaptation did not ought to be too difficult…

Chris Green

The AIP: maximising the picture

The Archaeological Investigations Project (AIP) is an ongoing project at the University of Bournemouth that collates the brief results of commercial archaeological investigations in England for each year.  With the very kind assistance of their Ehren Milner, we have extracted data from their database for our period in respect of two specific classes of monument type: settlements and field systems.  These two search queries were selected first as they seemed to me to be two of the key monument types of interest to our project: we will most probably ask Ehren to undertake further, different or revised queries in addition in the future.

The search terms used were taken from the English Heritage thesaurus, as that is used by many organisations and it seemed a sensible starting point.  Terms were selected from the list where they were relevant to our period of interest (e.g. we did not include terms under the settlement class like ‘Olympic village’ or ‘railway workers temporary settlement’), leaving out some longer terms which ought to be captured by a subset of their content (e.g. by including ‘field system’, we also ought to catch items named ‘Celtic field system’, ‘coaxial field system’, ‘enclosed field system’, etc.).

The field system query term list was:


The settlement query term list was:


It is not claimed that these lists are exhaustive or likely to capture every instance of these two classes of monument.  For instance, under settlement I was uncertain whether I should have included ‘villa’ or not, especially as I did include terms like ‘enclosure’ and ‘fort’.  The searches would also not capture objects recorded using any different terminologies, or sites of too small a size to be recorded within monument types of this sort of scale (e.g. an intervention which only discovered one ditch of a field system or settlement could not properly be recorded as such by the excavator without further supportive evidence, so would likely be recorded in the AIP as simply ‘ditch’).

In any event, subject to these caveats, the data received from Ehren (in .mdb database format) was exported to a .csv table and processed using a Python script to convert the OS NGRs (see my previous post) to numeric x and y coordinates (only 3 sites of over 4,000 seemed to have incorrectly recorded NGRs, i.e. they had an odd number of digits) and to extract the periods recorded for each site to a separate field in the new output table.  This processed table can then be imported into ArcGIS in the conventional fashion.

Here is a map of the results, plotted against our initial proposed case study locations:

AIP settlements and field systems (Bronze Age to early medieval)
Settlements and field systems (Bronze Age to early medieval) taken from AIP data and plotted against proposed case study areas.

This data is especially useful as it helps to fill gaps between areas where the National Mapping Programme (NMP) has not yet been undertaken.  For example, Cambridgeshire has had much commercial archaeology done in recent years, but has not been surveyed by the NMP as yet.  Hopefully, by using data from the HERs, EH data, PAS data, and AIP data in combination (alongside other more specific datasets), we ought to be able to build up an excellent picture of English archaeology for our period.

Chris Green