Search Your Data

This guide will show you how to search your data without decrypting it.

Key concepts in this guide:

• Token Search Indexes
• Expressions
• Token Search

Getting Started

To get started, you will need a Basis Theory account.

Sign Up

Sign In

Next you will need a Private Application in order to create and search tokens.

Click here to create a Private Application or login to your Basis Theory account and create a new application with the following settings:

• Name - Search Data Guide
• Application Type - Private
• Permissions - token:create, token:search

Save the API Key from the created Private Application as it will be used later in this guide to create and search the tokens.

Create Tokens

When you create a token, you can supply one or more Search Indexes, which are Expressions based on the Liquid template language. These Search Indexes are computed at the time of token creation to securely generate search indexes needed to enable searching.

Run the following in your terminal to create some tokens:

curl "https://api.basistheory.com/tokenize" \
-X "POST" \
-H "BT-API-KEY: <API_KEY>" \
-H "Content-Type: application/json" \
-d '[{
    "type": "token",
    "data": {
      "first_name": "Luke",
      "last_name": "Skywalker",
      "social_security_number": "111-22-3333",
      "email_address": "luke@skywalkerranch.com"
    },
    "mask": {
      "first_name": "{{ data.first_name }}",
      "last_name": "{{ data.last_name | slice: 0 }}",
      "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
      "email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
    },
    "metadata": {
      "customer_type": "vip",
      "spaceship": "X-Wing"
    },
    "containers": [
      "/customers/skywalkers/"
    ],
    "search_indexes": [
      "{{ data.first_name | downcase }}",
      "{{ data.last_name | downcase }}",
      "{{ data.social_security_number }}",
      "{{ data.social_security_number | last4 }}",
      "{{ data.email_address | downcase }}",
      "{{ data.email_address | split: '\''@'\'' | last }}"
    ]
}, {
    "type": "token",
    "data": {
      "first_name": "Han",
      "last_name": "Solo",
      "social_security_number": "444-55-6666",
      "email_address": "han.solo@skypilots.com"
    },
    "mask": {
      "first_name": "{{ data.first_name }}",
      "last_name": "{{ data.last_name | slice: 0 }}",
      "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
      "email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
    },
    "metadata": {
      "customer_type": "vip",
      "spaceship": "Millenium Falcon"
    },
    "containers": [
      "/customers/solo/"
    ],
    "search_indexes": [
      "{{ data.first_name | downcase }}",
      "{{ data.last_name | downcase }}",
      "{{ data.social_security_number }}",
      "{{ data.social_security_number | last4 }}",
      "{{ data.email_address | downcase }}",
      "{{ data.email_address | split: '\''@'\'' | last }}"
    ]
}, {
    "type": "token",
    "data": {
      "first_name": "Leia",
      "last_name": "Organa",
      "social_security_number": "777-88-9999",
      "email_address": "leia@skywalkerranch.com"
    },
    "mask": {
      "first_name": "{{ data.first_name }}",
      "last_name": "{{ data.last_name | slice: 0 }}",
      "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
      "email_address": "{{ data.email_address | split: '\''@'\'' | last }}"
    },
    "metadata": {
      "customer_type": "standard",
      "spaceship": "Tantive IV"
    },
    "containers": [
      "/customers/skywalkers/"
    ],
    "search_indexes": [
      "{{ data.first_name | downcase }}",
      "{{ data.last_name | downcase }}",
      "{{ data.social_security_number }}",
      "{{ data.social_security_number | last4 }}",
      "{{ data.email_address | downcase }}",
      "{{ data.email_address | split: '\''@'\'' | last }}"
    ]
}]'

Be sure to replace <API_KEY> with the Private API Key you created in the Getting Started step.

This will create several search indexes:

• Lowercase first name
• Lowercase last name
• Full social security number
• Last four of the social security number
• Lowercase email address
• Domain of the email address

Search Tokens

Now that we have some data tokenized, we can demonstrate several possible ways to search over the data. Basis Theory uses a Lucene-based query syntax to power the search engine.

Search by Data

Run the following command in your terminal to search for tokens having the indexed data value 3333, which we expect to match the last 4 digits of the social_security_number field on Luke Skywalker's token:

curl "https://api.basistheory.com/tokens/search" \
  -H "BT-API-KEY: <API_KEY>" \
  -H "Content-Type: application/json" \
  -X "POST" \
  -d '{
    "query": "data:3333",
    "page": 1,
    "size": 20
  }'

You should see a JSON response similar to:

{
  "pagination": {
    "total_items": 1,
    "page_number": 1,
    "page_size": 20,
    "total_pages": 1
  },
  "data": [
    {
      "id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
      "type": "token",
      "tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
      "data": {
        "first_name": "Luke",
        "last_name": "S",
        "social_security_number": "XXX-XX-3333",
        "email_address": "skywalkerranch.com"
      },
      "metadata": {
        "customer_type": "vip",
        "spaceship": "X-Wing"
      },
      "created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
      "created_at": "2022-12-21T20:49:11.6884577+00:00",
      "fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
      "fingerprint_expression": "{{ data | stringify }}",
      "mask": {
        "first_name": "{{ data.first_name }}",
        "last_name": "{{ data.last_name | slice: 0 }}",
        "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
        "email_address": "{{ data.email_address | split: '@' | last }}"
      },
      "search_indexes": [
        "{{ data.social_security_number }}",
        "{{ data.email_address | split: '@' | last }}",
        "{{ data.social_security_number | last4 }}",
        "{{ data.last_name | downcase }}",
        "{{ data.email_address | downcase }}",
        "{{ data.first_name | downcase }}"
      ],
      "containers": [
        "/customers/skywalkers/"
      ]
    }
  ]
}

Search by Metadata

Run the following command in your terminal to search for tokens having a metadata key customer_type with a value of vip:

curl "https://api.basistheory.com/tokens/search" \
  -H "BT-API-KEY: <API_KEY>" \
  -H "Content-Type: application/json" \
  -X "POST" \
  -d '{
    "query": "metadata.customer_type:vip",
    "page": 1,
    "size": 20
  }'

You should see a JSON response similar to:

{
  "pagination": {
    "total_items": 2,
    "page_number": 1,
    "page_size": 20,
    "total_pages": 1
  },
  "data": [
    {
      "id": "400394da-79d3-41f6-a45f-4e60c3e5b0a0",
      "type": "token",
      "tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
      "data": {
        "first_name": "Han",
        "last_name": "S",
        "social_security_number": "XXX-XX-6666",
        "email_address": "skypilots.com"
      },
      "metadata": {
        "customer_type": "vip",
        "spaceship": "Millenium Falcon"
      },
      "created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
      "created_at": "2022-12-21T20:49:11.6884577+00:00",
      "fingerprint": "Hi9TgzHi2otCqBeHKM5v9eAroGtUGUzTPFUCX3akRAEM",
      "fingerprint_expression": "{{ data | stringify }}",
      "mask": {
        "first_name": "{{ data.first_name }}",
        "last_name": "{{ data.last_name | slice: 0 }}",
        "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
        "email_address": "{{ data.email_address | split: '@' | last }}"
      },
      "search_indexes": [
        "{{ data.social_security_number | last4 }}",
        "{{ data.last_name | downcase }}",
        "{{ data.email_address | downcase }}",
        "{{ data.social_security_number }}",
        "{{ data.email_address | split: '@' | last }}",
        "{{ data.first_name | downcase }}"
      ],
      "containers": [
        "/customers/solo/"
      ]
    },
    {
      "id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
      "type": "token",
      "tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
      "data": {
        "first_name": "Luke",
        "last_name": "S",
        "social_security_number": "XXX-XX-3333",
        "email_address": "skywalkerranch.com"
      },
      "metadata": {
        "customer_type": "vip",
        "spaceship": "X-Wing"
      },
      "created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
      "created_at": "2022-12-21T20:49:11.6884577+00:00",
      "fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
      "fingerprint_expression": "{{ data | stringify }}",
      "mask": {
        "first_name": "{{ data.first_name }}",
        "last_name": "{{ data.last_name | slice: 0 }}",
        "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
        "email_address": "{{ data.email_address | split: '@' | last }}"
      },
      "search_indexes": [
        "{{ data.social_security_number }}",
        "{{ data.email_address | split: '@' | last }}",
        "{{ data.social_security_number | last4 }}",
        "{{ data.last_name | downcase }}",
        "{{ data.email_address | downcase }}",
        "{{ data.first_name | downcase }}"
      ],
      "containers": [
        "/customers/skywalkers/"
      ]
    }
  ]
}

Perform Compound Search

Run the following command in your terminal to search for tokens having both a metadata key customer_type with a value of vip and a metadata key spaceship with a value of X-Wing:

curl "https://api.basistheory.com/tokens/search" \
  -H "BT-API-KEY: <API_KEY>" \
  -H "Content-Type: application/json" \
  -X "POST" \
  -d '{
    "query": "metadata.customer_type:vip AND metadata.spaceship:X-Wing",
    "page": 1,
    "size": 20
  }'

You should see a JSON response similar to:

{
  "pagination": {
    "total_items": 1,
    "page_number": 1,
    "page_size": 20,
    "total_pages": 1
  },
  "data": [
    {
      "id": "5855f821-a207-4f07-8204-4b7c96cb2bbc",
      "type": "token",
      "tenant_id": "dd19bffb-db97-4ef7-8630-54834a7c1243",
      "data": {
        "first_name": "Luke",
        "last_name": "S",
        "social_security_number": "XXX-XX-3333",
        "email_address": "skywalkerranch.com"
      },
      "metadata": {
        "customer_type": "vip",
        "spaceship": "X-Wing"
      },
      "created_by": "f1d47dd9-34ac-407c-bb0c-99ad786f9167",
      "created_at": "2022-12-21T20:49:11.6884577+00:00",
      "fingerprint": "4wPsbGizTkqeh3ZGXFEbPep5zmf6ffokX37SDEAw9m2S",
      "fingerprint_expression": "{{ data | stringify }}",
      "mask": {
        "first_name": "{{ data.first_name }}",
        "last_name": "{{ data.last_name | slice: 0 }}",
        "social_security_number": "{{ data.social_security_number | reveal_last: 4 }}",
        "email_address": "{{ data.email_address | split: '@' | last }}"
      },
      "search_indexes": [
        "{{ data.social_security_number }}",
        "{{ data.email_address | split: '@' | last }}",
        "{{ data.social_security_number | last4 }}",
        "{{ data.last_name | downcase }}",
        "{{ data.email_address | downcase }}",
        "{{ data.first_name | downcase }}"
      ],
      "containers": [
        "/customers/skywalkers/"
      ]
    }
  ]
}

Conclusion

Utilizing Token Search empowers you to provide business functionality against your encrypted data without the need to decrypt it, enabling things like reporting and customer service operations.

Learn More

• What is Token Search?
• Access Controls
• Display Masked Data