Water Quality Archive Documentation

Introduction

The Water Quality Archive holds data on water quality monitoring carried out by the Environment Agency as well as some self-monitoring data provided to the Environment Agency by external customers. Samples are taken from sampling points round the country and then analysed by laboratories to measure aspects of the water quality or the environment at the sampling point. The archive provides data on these measurements and samples dating from 2000 to present day. It contains 58 million measurements on nearly 4 million samples from 58 thousand sampling points.

The archive provides an API to allow selective access to the data, together with the ability to download the data split into either pre-defined or customizable subsets. The data is made available in CSV, JSON and RDF formats.

Licence

This archive is provided as open data under the Open Government Licence with no requirement for registration. If you make use of this data please acknowledge this with the following attribution statement:

uses Environment Agency water quality data from the Water Quality Archive (Beta)

API Summary

This is a brief summary of the APIs available. In the links in the tables {base} corresponds to http://environment.data.gov.uk/water-quality.

Sampling points

Samples

Measurements

Reference data

Miscellaneous

The query APIs are subject to a timeout limit, currently set to two minutes. When requesting a very large number of measurements, or measurements selected by a complex query, then the alternative batch request mechanism can be used.

Introduction to the data

Sampling points

These are the locations from which samples are taken. Each has a name (a short label and a longer descriptive comment) and a geographic location (available in both British National Grid and lat/long versions) and is associated with some operational area of the Environment Agency. A list of these areas is available as a map or as data. These areas are used to break up the data into manageable downloadable pieces. There are various types of sampling points.

The archive contains data from sampling points which were in use between 2000 and 2015. Some of these sampling points are no longer in use in which case they will have a status of closed.

Samples

Samples are taken from a sampling point at some date/time, for some defined purpose. The rate of sampling varies widely - some points may be sampled very regularly as part of routine monitoring, others may only be sampled as a result of some specific event. There are many purposes for which a sample can be taken but they broadly divide into either checking compliance with some permit or general monitoring. This is indicated in the data with an isComplianceSample flag.

The material sampled is also indicated. There are many sampled material types possible and these are not limited to the water itself but include associated air, soil and biological samples.

Measurements and Determinands

A sample is analysed to determine one or more properties of the sample or of the associated environment. The properties measured are formally called determinands. In fact the determinand defines not just what has been measured (e.g. dissolved oxygen) but the units the result is expressed in and, implicitly, how the measurement was carried out.

There are a very large number of determinands used in the data (over 7000). To simplify working with these we have created some groupings of useful determinands. It is also possible to query to find out what determinands have ever been measured on samples from a particular sampling point.

The result of a measurement is normally a number, expressed in the units of the determinand. For example, measurement AN-1770585-0237 gives a value of 7.62 mg/l of Magnesium in a surface water sample from Marley Gap in 2014.

However, sometimes a measurement is only an upper or lower bound. For example, the laboratory test may not be able to detect the substance if it is below some concentration. In such cases the measurement will have an associated qualifier. For example, AN-1805502-0119 represents a measurement of less than 0.00059 mg/l of Ammonia (un-ionised as N) on a sample in 2015.

For some determinands the resulting measurement is not a number but a coded value. For example, measurement: MD-1502966-1181 indicates that the weather was Hot (level 5 on a 6 point scale of qualitative weather temperature from Very cold to Very hot) at the time the sample was taken. The example also illustrates that measurements may apply to the environment of the sampling point at the time of the sample, and are not restricted to properties of the sample itself.

API Structure

The APIs provide a REST style access to the data via simple HTTP GET requests which return data in JSON, CSV or RDF formats.

Simple requests

For example fetching data from: http://environment.data.gov.uk/water-quality/id/sampling-point will return a JSON data packet contain a list of some Sampling points:

{
    "@context": "http://environment.data.gov.uk/water-quality/doc/context-TBD.jsonld",
    "meta": {
        "publisher": "Environment Agency",
        "licence": "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/",
        "documentation": "http://environment.data.gov.uk/water-quality/view/doc/reference",
        "version": "0.1",
        "comment": "WARNING: Pre-alpha test service",
        "hasFormat": [
            "http://environment.data.gov.uk/water-quality/id/sampling-point.csv",
            "http://environment.data.gov.uk/water-quality/id/sampling-point.rdf",
            "http://environment.data.gov.uk/water-quality/id/sampling-point.ttl",
            "http://environment.data.gov.uk/water-quality/id/sampling-point.html" ],
        "limit": 50
    },
    "items": [
        {
            "@id": "http://environment.data.gov.uk/water-quality/id/sampling-point/AN-WOODTON",
            "area": {
                "@id": "http://environment.data.gov.uk/registry/def/ea-organization/ea_areas/1-2",
                "label": "Essex Norfolk and Suffolk"
            },
            "comment": "WOODTON STW FINAL EFFLUENT",
            "easting": 630030,
            "label": "WOODTON STW F/E",
            "lat": 52.492125,
            "long": 1.387003,
            "northing": 293640,
            "samplingPointStatus": {
                "@id": "http://environment.data.gov.uk/water-quality/def/sampling-point-status/open",
                "label": "open"
            },
            "samplingPointType": {
                "@id": "http://environment.data.gov.uk/registry/def/water-quality/sampling_point_types/SA",
                "label": "SEWAGE DISCHARGES - FINAL/TREATED EFFLUENT - WATER COMPANY"
            },
            "subArea": {
                "@id": "http://environment.data.gov.uk/water-quality/id/ea-subarea/1-2-H",
                "label": "NORFOLK TEAM SUB AREA"
            }
        } , ...
    ]
}

The returned JSON data from all API endpoints follows the same structure of three elements: context, metadata and item(s).

The @context reference is placeholder that will, in due course, enable the JSON data to be read as json-ld.

Metadata and versioning

The metadata block:

"meta": {
    "publisher": "Environment Agency",
    "licence": "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/",
    "documentation": "http://environment.data.gov.uk/water-quality/view/doc/reference",
    "version": "0.1",
    "comment": "WARNING: Pre-alpha test service",
    "hasFormat": [
        "http://environment.data.gov.uk/water-quality/id/sampling-point.csv",
        "http://environment.data.gov.uk/water-quality/id/sampling-point.rdf",
        "http://environment.data.gov.uk/water-quality/id/sampling-point.ttl",
        "http://environment.data.gov.uk/water-quality/id/sampling-point.html" ]
    "limit" : 50
}

provides information on the publisher and applicable licence as well as a link to this, or other, documentation. If the resource is also available in other formats then hasFormat will supply list of URLs for those alternative formats (the media type is implied by a suffix, see content types).

The metadata block also includes version number information. The intention is that updates to the API should maintain backward compatibility. If an incompatible change to the API is required then we will attempt to provide access to the prior version for a transitional period. In that case the meta block will also provide replaces and isReplacedBy links between the new and the old versions of the affected API endpoints.

Finally, in the case of calls which provide lists of results then any applied limit to the length of the list, and offset from the start of the list, will be shown in the metadata as limit and offset values.

Items and identifiers

The items element in the JSON response will contain either a description of a single item or an array of items. For API endpoints which return lists of items (e.g. all sampling points) then value will always be an array even if the list only has one entry. For API endpoints which describe a specific item (e.g. a particular sampling point) the value will always be an object, with no wrapping array.

Each item will normally be identified by a URI given in the @id field. So in the earlier example:

{
    "@id": "http://environment.data.gov.uk/water-quality/id/sampling-point/AN-WOODTON",
    "area": {
        "@id": "http://environment.data.gov.uk/registry/def/ea-organization/ea_areas/1-2",
        "label": "Essex Norfolk and Suffolk"
    },
    "comment": "WOODTON STW FINAL EFFLUENT",
    "samplingPointStatus": {
        "@id": "http://environment.data.gov.uk/water-quality/def/sampling-point-status/open",
        "label": "open"
    },
    ...
}

we saw that the first sampling point has a URI identifier of:

The items that the sampling point links to, such as the Environment Agency area are also given URIs. In that case the URI is defined in the Environment registry and the same identifier is used in other places, not just the Water Quality Archive:

Similarly codes from code lists to indicate things like status, purpose, materials, determinands and so on are all given URIs along with descriptive labels.

When one item references another, such as in this case, the API will typically include key attributes of the referenced item in-line for convenience. However, we can always fetch (GET) data from the reference item URI to obtain a full description, in a variety of formats.

The URIs follow a standard pattern where real identifiable objects or places in the world, such as sampling points, start with id, abstract definitions, such as codes, start with def and real data such as records of samples and measurements start with data.

The API also follows common conventions where a URI of an item in a collection is an extension of the URI for a collection. So that the URI for the collection of all sampling points is:

and the URI for a particular sampling point is:

Content Types

The descriptions of individual items can be obtained in multiple formats. The default format is JSON but all item descriptions can also return information in RDF formats (RDF/XML and Turtle). In addition items offer a simple HTML rendering which makes it possible to look at these reference identifiers conveniently in a browser. The HTML rendering is intended as an aid to help developers understand the data, it is not an end-user application.

Similarly lists of items can be obtained in multiple formats. The default is JSON (in which the items field will contain an array of objects) but they are also available in CSV and RDF formats (RDF/XML and Turtle)

To select the desired format use standard HTTP content negation. E.g.:

curl -i -H "Accept: text/csv" http://environment.data.go.uk/water-quality/id/sampling-point?search=clifton

All endpoints also support a short cut of appending a type suffix to the URI to force a particular content type. The supported suffixes are:

The media type text/html is handled in a special way. HTML is only returned if that is the only media type acceptable (through use of the .html suffix or because that's the only option given in content negotiation accept header). This means that by default you see machine readable data (JSON) even in a web browser (which normally prioritizes HTML over anything else). While this handling is non-standard, feedback (on similar APIs) suggested this is a pragmatically convenient behaviour.

Lists: filtering, paging, sorting, views

Some endpoints return information describing a single identified item but many return information on a list of items. In that case the value of the items field is an array of items. Such list endpoints support query parameters to filter the list to only include some items, for example:

This returns all measurements from the sampling point AN-26M19 filtered to only include those from the year 2014 and to only include measurements from the pharmaceuticals group of determinands.

The specific filters available depend on the endpoint and are documented below. Though many endpoints will support a generic text search option search=x which will filter the list for only those whose label or comment fields contain a matching word. Multiple words can be used. For example, the following searches the list of determinands for ones that mention both temperature and water:

The search keywords will be filtered to remove punctuation characters meaningful to the underlying search engine, so only plain alpha-numberic keywords can be reliably used as search terms.

In cases where the value you want to filter on is a URI then there is a shortcut available. Consider the case of searching the list of determinands for those that are measured in a particular unit. The unit given below represents mg/l:

 {
     "@id" : "http://environment.data.gov.uk/water-quality/def/units/0205" ,
     "code" : "0205" ,
     "comment" : "MILLIGRAM PER LITRE" ,
     "inScheme" : {
         "@id" : "http://environment.data.gov.uk/water-quality/def/units/scheme"
      } ,
      "label" : "mg/l" ,
      "prefLabel" : "mg/l" ,
      "type" : [
          { "@id" : "http://qudt.org/schema/qudt#Unit" },
          { "@id" : "http://www.w3.org/2004/02/skos/core#Concept" }
      ]
  }

So the full identifier for the unit is the URI http://environment.data.gov.uk/water-quality/def/units/0205 which can be used to filter the determinand list for those measured in mg/l.

However, the API will also accept a short cut of just the last segment of the URI, 0205 in this case. So that the shorter query is:

It is also possible to achieve the same effect by filtering on the label of the unit, though normally the URI (full or truncated) is the better option:

When filtering on simple literal values such as numbers, strings or true/false then just use those values in the filter. For the case of dates then the date format must follow ISO format, for example 2015-03-01.

In most cases it is possible to apply multiple filters on the sample parameter. This corresponds to a disjunction, a request for information matching either filter value. For example, the follow request lists samples from two sampling points (for the year 2015):

In addition to filtering then list endpoints support a number of view modifiers to control the number of results returned, or their order. All such parameters which affect the presentation of the data but not which items are returned are given a prefix of "_". The main modifiers are:

If a limit or offset is applicable, whether explicit in the query or implicitly imposed by the API, then the metadata object will include a limit or offset field to show what limits were applied.

The _sort is carried out on the server before any limits/offets are applied. So these modifiers together allow you to obtain the first or last few of a longer list. However, the sort options are expense and may lead queries to exceed the query time-out limit. For this reason it is often better to simply request the whole slice of data of interest and then reorder locally for presentation or analysis.

The _view modifier controls how much information is return about the items. Most list endpoints support views of compact, full or default, with the default view being used if no view is explicitly specified.

Sampling points

These are the locations from which samples are taken. Each has a name (a short label and a longer descriptive comment) and a geographic location (available in both British National Grid and lat/long versions) and is associated with some operational area of the Environment Agency.

The API for listing sampling points is:

Each sampling point has its own URI from which you can retrieve more information:

Results returned

Samples

Samples are taken from a sampling point at some date/time, for some defined purpose. The rate of sampling varies widely - some points may be sampled very regularly as part of routine monitoring, others may only be sampled as a result of some specific event. There are many purposes for which a sample can be taken but they broadly divide into either checking compliance with some permit or general monitoring.

The API for listing all samples is:

Or to just list samples from a particular sampling point:

To count the number of samples available per-year from a sampling use:

This call will only return information in JSON format and the return items will be objects with year and count values.

Each sample has its own URI from which you can retrieve more information:

It is also possible to list all the measurements on a particular sample:

Results returned

Measurements

The measurements are either the results of analysing a sample to determine one or more properties, or they can be properties of the associated environment (such as temperature at the time the sample was taken). The properties measured are more properly called determinands, A determinand defines not just what has been measured (e.g. dissolved oxygen) but the units the result is expressed in and, implicitly, how the measurement was carried out.

The result of a measurement is normally a number, expressed in the units of the determinand. For example, measurement AN-1770585-0237 gives a value of 7.62 mg/l of Magnesium in a surface water sample from Marley Gap in 2014.

However, sometimes a measurement is only an upper or lower bound. For example, the laboratory test may not be able to detect the substance if it is below some concentration. In such cases the measurement will have an associated qualifier. For example, AN-1805502-0119 represents a measurement of less than 0.00059 mg/l of Ammonia (un-ionised as N) on a sample in 2015.

For some determinands the resulting measurement is not a number but some coded value. For example, measurement: MD-1502966-1181 indicates that the weather was Hot (level 5 on a 6 point scale of qualitative weather temperature from Very cold to Very hot) at the time the sample was taken.

The API for listing all measurements is:

Or to just list measurements from a particular sampling point:

or from a particular sample:

Each measurement has its own URI from which you can retrieve the information:

Results returned

Batch queries

The query APIs are subject to a time-out limit, currently set to two minutes. The batch API provides an alternative query mechanism for obtaining measurements. The API:

This takes the same filter parameters as the /water-quality/data/measurement API. If that set of measurements has previously been generated then the call will return (a redirection to) a cached copy of the results (currently in CSV format only). If no one has requested that particular set of measurements already then a new batch request will be scheduled on a batch processing queue and the call will return (a redirection to) a status page which shows the status of that request in the queue. The status page is available in JSON and HTML format. Repeated fetches of the status page will show the current progress of the request. The queries in the batch processing queue are not subject to time-out so large queries or query results can be accommodated.

The request status in JSON looks like:

{
    "key" : "CB-2003" ,
    "status" : "InProgress" ,
    "positionInQueue" : 0 ,
    "eta" : 86333 ,
    "started" : "2/6/16 8:36 PM"
}

Where the status can be one of Pending InProgress Completed or Failed. The eta element shows the expected time to completion - this is given in ms though the estimate is very approximate and may be inaccurate by minutes or more.

When the batch processing has completed then the status response will change to show a URL from which the result can be download:

{
    "key": "CB-2003",
    "status": "Completed",
    "url": "https://s3-eu-west-1.amazonaws.com/environment-open-data/wims/cache/CB-2003.csv"
}

In addition, it is also possible to POST a query to the same batch API. The filter settings can still be provided in the query parameters, just as for GET requests, but in addition it is possible to send the filter parameters in the body of the POST request in application/x-www-form-urlencoded format. This allows larger sets of filters values to be sent, for example in order to request information from a large number of specific sampling points.

The result of a successful batch query POST submission will be a 201 (Created) response with a location header giving the URL of the status information on the request, in the same format as the above.

Determinands and units

Determinands identify a property which can be measured on a sample or the sampling environment, together with the units in which the result of that measurement will be expressed. There are over 7000 such determinands within the Water Quality Archive. A number of API calls are provided to assist with finding determinands:

Results returned - determinands

Results returned - units

Other reference information

A number of other codes or entities are used in the data and the set of legal values for these can be listed using the the follow API calls.

These all support filtering by text search using the search={x} parameter.

Data reference