Agrimetrics Data Platform

Cursors

Cursors allow you to efficiently retrieve a page of results with each request, and to progress through subsequent pages until all data has been returned, even if the total data to be retrieved is very large.

By default, our GraphQL API will attempt to return you as many results as can be retrieved, processed in memory, and returned in a response in a reasonable amount of time. For queries that result in large result sets that cannot be returned within such constraints, it will be necessary to understand and use cursors.

Unless you use cursors for the types of queries that support them, you will not know if you have retrieved the entire result set for your query.

Cursor properties

  • cursor - an output property containing an opaque value used to fetch the next page of results. This value is null in the event there is no further data.
  • after - an input parameter that is used to pass the cursor from a previous page of results, in order to fetch the next.
  • first - an input parameter that is used to limit the number of results returned in each page.

Requesting Cursors

The process requires you request a cursor property in your first request, then pass it to the next request as an after input property, requesting the next cursor for you to pass, and repeating this process until the cursor returned is null. Generally, it is useful to use parameterized queries to do this.

Here is a version of our simple field query example, but here asking for the area of fields in a radius of 100,000m from a location, 5 results at a time (because of first: 5), parameterized so we can use a cursor to get all the results:

query($cursor: String) {
  fields(geoFilter: {distance: {LE: 100000}, location: {type: Point, coordinates: [0.089372, 52.245591]}}, first: 5, after: $cursor) {
    id
    area {
      value
    }
    cursor
  }
}

The cursor parameter, after, is an optional String, so we can omit it from our variables or pass null on the first call. We request the next fields.cursor along with any other properties we require. We'd POST to the GraphQL API with a JSON body containing query and variables properties like this:

{ 
  "query": "query($cursor: String) {fields(geoFilter: {distance: {LE: 100000},location: {type: Point, coordinates: [0.089372,52.245591]}}, first: 5, after: $cursor){id area {value} cursor}}",
  "variables": {
    "cursor": null
  }
}

This would return a result like, containing a cursor for each field:

{
  "data": {
    "fields": [
      ...,
      {
        "id": "https://data.agrimetrics.co.uk/fields/--1ASl2R8_WkoVr4oBQVUA",
        "area": {
          "value": 5.2459
        },
        "cursor": "VGhpcyBpcyBhbiBleGFtcGxlIGN1cnNvcgo="
      }
    ]
  }
}

On the next call, we would extract the cursor property from the last element of the previous fields array, and, if it is not null, pass it as the value of the cursor input variable, so it populates the after parameter in the next query:

{ 
  "query": "query($cursor: String) {fields(geoFilter: {distance: {LE: 100000},location: {type: Point, coordinates: [0.089372,52.245591]}}, first: 5, after: $cursor){id area {value} cursor}}",
  "variables": {
    "cursor": "VGhpcyBpcyBhbiBleGFtcGxlIGN1cnNvcgo="
  }
}

Specifying the cursor retrieves the next 5 fields, each of which have its own distinct cursor, the last of which can be used to get the next page of results. Repeat this process until the last field has a value of null for cursor.

You can actually fetch the next page of results using any previous cursor, but it is not normally useful to do so, since the next page of results will overlap with the previous.

Nested cursors

As well as cursoring over fields, some properties, such as weatherObservations have their own cursor, in this case, one that pages over a time series.

In this case, assuming that the initial query returns non-null cursors, we'll need to page over both the fields, and the weather observations, so we'll have to keep track of both cursors. We'd use two input parameters to separately keep track of each cursor, looping over the inner cursor until it becomes null, before fetching the next page of fields using the outer cursor; we'd repeat this until the nexy outer cursor also becomes null.

See our Examples Repository for code that uses multiple cursors to get rainfall data for a large area.