Skip to content

Metadata

Metadata helps you store additional information about a document, often referred to as document attributes. It provides added context and enables more refined, targeted searches via powerful filtering, sorting, and faceting capabilities.

Gainly offers you a flexible way to store document metadata in the form of arbitrary key-value pairs (can be nested) with the following limits.

Example 
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Metadata limits

You can add 100 total key-value pairs:

  • Key: 50 character limit.
  • Value: 500 character limit.

Metadata schema

Metadata schema refers to the structured definition of the metadata keys. It defines the key names and data types for the values.

Automatically inferred

Gainly automatically infers the data type of a key based on its value the first time the key is submitted. Thus Gainly automatically infers your metadata schema as you add metadata to your documents.

Special Case: Geo-Point Fields

Keys ending in _geo_point will be inferred as geo_point fields that accept a geographic point. Thus geo_point fields are inferred by the key name.

Metadata schema endpoint

You can retrieve your automatically-inferred metadata schema using the Metadata Schema endpoint.

Metadata schema in Test vs. Live modes

The metadata schema for your Test mode is separate from that of your Live mode. This allows you to test freely without impacting the metadata schema in your live mode.

Best practices

  1. Define your metadata schema early:

    • It's a good practice to define your metadata schema early on so that your team has a consistent structure to work with.
    • This should include predefined key names and data types for values.
    • Only add metadata to documents in live mode after finalizing your metadata schema.

  2. Validate metadata against your schema:

    • Always validate metadata against your schema before passing it to the metadata parameter in Add document or Update document API requests.

Cannot modify or delete existing metadata keys

To support high-performance searches including filtering, sorting, and faceting - each metadata key is implemented as a field in the semantic index.

To ensure index consistency, the semantic index fields are immutable, meaning you cannot modify (rename or change data type) or delete an existing metadata key.

However you can do the following whenever needed:

  • Change the value of a metadata key in a document.
  • Add new metadata keys (and their values) to documents.

Professional Services

If modifying or deleting a key is absolutely essential, Gainly Professional Services team can assist you for a fee. Please note this process will involve some downtime for your semantic index.

Supported field types

The following field types are supported for keys in your metadata:

String

A string sequence of characters. Please note these are not used for text search - but rather used as a part of filter, sort, and facet.

IMPORTANT: Will appear as field type of keyword

String fields will appear as keyword in your metadata schema, meaning the field is not indexed for full-text search. Instead, it is used for filtering, sorting, and faceting.

In the following example, color will be inferred as a string and assigned a field type of keyword:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

String fields will appear in metadata schema as shown below:

"color": {
  "type": "keyword"
}

Date

A string in ISO 8601 format, which can include date, time, and timezone.

In the following example, the highlighted fields will be inferred as date fields:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Integer

A whole number (without decimals) that can range from negative to positive values, between -231 and 231 - 1.

In the following example, length will be inferred as an integer field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Float

A number that can contain decimals, allowing for both positive and negative values (single precision 32-bit IEEE 754 floating-point value).

In the following example, price will be inferred as a float field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Boolean

Accepts true or false values. Empty string passed to a boolean field will be treated as false.

In the following example, is_sale will be inferred as a boolean field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Geo-point

A geographic point specified by latitude and longitude coordinates.

IMPORTANT: Use key names ending in _geo_point

Use key names ending in _geo_point to make sure the key is inferred as a geo-point field. Unlike other field types, geo_point fields are inferred by their key names.

In the following example, location_geo_point will be inferred as a geo_point field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Arrays

Arrays can only store values of one type - such as an array of integers, an array of strings, etc.

In the following example, tags will be inferred as an array of strings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Arrays will be represented in the metadata schema by the field type of an individual element within that array:

"tags": {
  "type": "keyword"
}

Objects

Objects are standard JSON objects, which can have fields of their own. This means you're able to implement nested fields in your metadata.

In the following example, sale will be inferred as an object field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
"metadata": {
  "color": "red",
  "is_sale": true,
  "length": 52,
  "location_geo_point": {
      "lat": 45.67,
      "lon": -111.04
  },
  "price": 99.99,
  "released_on": "2024-01-01T00:00:00.000Z",
  "sale": {
      "sale_price": 79.99,
      "sale_starts": "2024-09-01T00:00:00.000Z",
      "sale_ends": "2024-10-01T00:00:00.000Z"
  },
  "tags": ["shirt", "clothing"]
}

Object fields appear in metadata schema as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
"sale": {
  "properties": {
    "sale_ends": {
      "type": "date",
      "format": "strict_date_optional_time"
    },
    "sale_price": {
      "type": "float"
    },
    "sale_starts": {
      "type": "date",
      "format": "strict_date_optional_time"
    }
  }
}

Add metadata

You can add metadata to a document using the metadata parameter in the following endpoints:

Update metadata

You can update the metadata for a document using the metadata parameter of Update Document endpoint. Specifically, you can:

  • Add a new key-value pair to a document
  • Update the value of an existing key in a document
  • Unset an existing key from a document
  • Set metadata to null for a document

However, modifying or deleting an existing key is not supported.

Metadata in queries

You can use metadata in your API requests to perform:

While performing these operations, specify your metadata fields using dot notation:

  • metadata.color
  • metadata.sale.sale_starts