3.4.4. Index Module

In this chapter, you'll learn about the Index Module and how you can use it.

What is the Index Module?#

The Index Module is a tool to perform high-performance queries across modules, for example, to filter linked modules.

While modules share the same database by default, Medusa isolates modules to allow using external data sources or different database types.

So, when you retrieve data across modules using Query, Medusa aggregates the data coming from different modules to create the end result. This approach limits your ability to filter data by linked modules. For example, you can't filter products (created in the Product Module) by their brand (created in the Brand Module).

The Index Module solves this problem by ingesting data into a central data store on application startup. The data store has a relational structure that enables efficient filtering of data ingested from different modules (and their data stores). So, when you retrieve data with the Index Module, you're retrieving it from the Index Module's data store, not the original data source.

Ingested Data Models#

All core data models in Medusa are ingested, including Product, Price, SalesChannel, and more. You can also index custom data models if they are linked to an ingested data model. You'll learn more about this in the Ingest Custom Data Models section.

How to Install the Index Module#

To install the Index Module, run the following command in your Medusa project to install its package:

npm
yarn
pnpm

Then, add the Index Module to your Medusa configuration in medusa-config.ts:

medusa-config.ts
1module.exports = defineConfig({2  // ...3  modules: [4    // ...5    {6      resolve: "@medusajs/index",7    },8  ],9})

Finally, run the migrations to create the necessary tables for the Index Module in your database:

Terminal
❯npx medusa db:migrate

Ingest Data#

The Index Module only ingests data when you start your Medusa server. So, to ingest the currently supported data models, start the Medusa application:

npm
yarn
pnpm

The ingestion process may take a while if your product catalog is large. You'll see the following messages in the logs:

Terminal
❯info:    [Index engine] Checking for index changes❯info:    [Index engine] Found 7 index changes that are either pending or processing❯info:    [Index engine] syncing entity 'ProductVariant'❯info:    [Index engine] syncing entity 'ProductVariant' done (+38.73ms)❯info:    [Index engine] syncing entity 'Product'❯info:    [Index engine] syncing entity 'Product' done (+18.21ms)❯info:    [Index engine] syncing entity 'LinkProductVariantPriceSet'❯info:    [Index engine] syncing entity 'LinkProductVariantPriceSet' done (+33.87ms)❯info:    [Index engine] syncing entity 'Price'❯info:    [Index engine] syncing entity 'Price' done (+22.79ms)❯info:    [Index engine] syncing entity 'PriceSet'❯info:    [Index engine] syncing entity 'PriceSet' done (+10.72ms)❯info:    [Index engine] syncing entity 'LinkProductSalesChannel'❯info:    [Index engine] syncing entity 'LinkProductSalesChannel' done (+11.45ms)❯info:    [Index engine] syncing entity 'SalesChannel'❯info:    [Index engine] syncing entity 'SalesChannel' done (+7.00ms)

Update Index on Data Changes#

The Index Module automatically updates its data store when data in the ingested data models change. So, you don't need to do anything to keep the data in sync.

For example, if you create a new product, the Index Module will ingest it into its data store.

Enable Index Module Feature Flag#

Since the Index Module is still experimental, the /store/products and /admin/products API routes will use the Index Module to retrieve products only if the Index Module's feature flag is enabled. By enabling the feature flag, you can filter products by their linked data models in these API routes.

To enable the Index Module's feature flag, add the following line to your .env file:

Code
MEDUSA_FF_INDEX_ENGINE=true

If you send a request to the /store/products or /admin/products API routes, you'll receive the following response:

Code
1{2  "products": [3    // ...4  ],5  "count": 2,6  "estimate_count": 2,7  "offset": 0,8  "limit": 509}

Notice the estimate_count property, which is the estimated total number of products in the database. You'll learn more about it in the Pagination section.

How to Use the Index Module#

The Index Module adds a new index method to Query and it has the same API as the graph method.

For example, to filter products by a sales channel ID:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { data: products } = await query.index({13    entity: "product",14    fields: ["*", "sales_channels.*"],15    filters: {16      sales_channels: {17        id: "sc_123",18      },19    },20  })21
22  res.json({ products })23}

This will return all products that are linked to the sales channel with the ID sc_123.

The index method accepts an object with the same properties as the graph method's parameter:

• entity: The data model's name, as specified in the first parameter of the model.define method used for the data model's definition.
• fields: An array of the data model’s properties, relations, and linked data models to retrieve in the result.
• filters: An object with the filters to apply on the data model's properties, relations, and linked data models that are ingested.

How to Ingest Custom Data Models#

Aside from the core data models, you can also ingest your own custom data models into the Index Module. You can do so by defining a link between your custom data model and one of the core data models, and setting the filterable property in the link definition.

For example, assuming you have a Brand Module with a Brand data model (as explained in the Customizations), you can ingest it into the Index Module using the filterable property in its link definition to the Product data model:

src/links/product-brand.ts
1import BrandModule from "../modules/brand"2import ProductModule from "@medusajs/medusa/product"3import { defineLink } from "@medusajs/framework/utils"4
5export default defineLink(6  {7    linkable: ProductModule.linkable.product,8    isList: true,9  },10  {11    linkable: BrandModule.linkable.brand,12    filterable: ["id", "name"],13  }14)

The filterable property is an array of property names in the data model that can be filtered using the index method. When the filterable property is set, the Index Module will ingest into its data store the custom data model.

But first, you must run the migrations to sync the link, then start the Medusa application:

npm
yarn
pnpm

You'll then see the following message in the logs:

Terminal
❯info:    [Index engine] syncing entity 'LinkProductProductBrandBrand'❯info:    [Index engine] syncing entity 'LinkProductProductBrandBrand' done (+3.64ms)❯info:    [Index engine] syncing entity 'Brand'❯info:    [Index engine] syncing entity 'Brand' done (+0.99ms)

You can now filter products by their brand, and vice versa. For example:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { data: products } = await query.index({13    entity: "product",14    fields: ["*", "brand.*"],15    filters: {16      brand: {17        name: "Acme",18      },19    },20  })21
22  res.json({ products })23}

This will return all products that are linked to the brand with the name Acme. For example:

Example Response
1{2  "products": [3    {4      "id": "prod_123",5      "brand": {6        "id": "brand_123",7        "name": "Acme"8      },9      // ...10    }11  ]12}

Apply Pagination with the Index Module#

Similar to Query's graph method, the Index Module accepts a pagination object to paginate the results.

For example, to paginate the products and retrieve 10 products per page:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { 13    data: products,14    metadata,15  } = await query.index({16    entity: "product",17    fields: ["*", "brand.*"],18    filters: {19      brand: {20        name: "Acme",21      },22    },23    pagination: {24      take: 10,25      skip: 0,26    },27  })28
29  res.json({ products, ...metadata })30}

The pagination object accepts the following properties:

• take: The number of items to retrieve per page.
• skip: The number of items to skip before retrieving the items.

When the pagination property is set, the index method will also return a metadata property. metadata is an object with the following properties:

• skip: The number of items skipped.
• take: The number of items retrieved.
• estimate_count: The estimated total number of items in the database matching the query. This value is retrieved from the PostgreSQL query planner rather than using a COUNT query, so it may not be accurate for smaller data sets.

For example, this is the response returned by the above API route:

Example Response
1{2  "products": [3    // ...4  ],5  "skip": 0,6  "take": 10,7  "estimate_count": 1008}

Cache Index Module Results#

You can cache Index Module results to improve performance and reduce database load. To do that, you can pass a cache property in the second parameter of the query.index method.

For example, to enable caching for a query:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true7  }8})

In this example, you enable caching of the query's results. The next time the same query is executed, the results are returned from the cache instead of querying the database.

Cache Properties#

cache is an object that accepts the following properties:

Set Cache Key#

By default, the Caching Module generates a cache key for a query based on the arguments passed to query.index. The cache key is a unique key that the cached result is stored with.

Alternatively, you can set a custom cache key for a query. This is useful if you want to manage invalidating the cache manually.

To set the cache key of a query, pass the cache.key option:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    key: "products-123456",8    // to disable auto invalidation:9    // autoInvalidate: false,10  }11})

In the example above, you cache the query results with the products-123456 key.

You can also pass a function as the value of cache.key:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    key: async (args, cachingModuleService) => {8      return await cachingModuleService.computeKey({9        ...args,10        prefix: "products"11      })12    }13  }14})

In the example above, you pass a function to key. It accepts two parameters:

1. The arguments of query.index passed as an array.
2. The Caching Module's service.

You generate the key using the computeKey method of the Caching Module's service. The query results will be cached with that key.

Set Cache Tags#

By default, the Caching Module generates relevant tags for a query based on the entity and its retrieved relations. Cache tags are useful to group related items together, allowing you to retrieve or invalidate items by common tags.

Alternatively, you can set the cache tags of a query manually. This is useful if you want to manage invalidating the cache manually, or you want to group related cached items with custom tags.

To set the cache tags of a query, pass the cache.tags option:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    tags: ["Product:list:*"],8  }9})

In the example above, you cache the query results with the Product:list:* tag.

You can also pass a function as the value of cache.tags:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    tags: (args) => {8      const collectionId = args[0].filter?.collection_id9      return [10        ...args,11        collectionId ? `ProductCollection:${collectionId}` : undefined,12      ]13    },14  }15})

In the example above, you use a function to determine the cache tags. The function accepts the arguments passed to query.index as an array.

Then, you add the ProductCollection:id tag if collection_id is passed in the query filters.

Set TTL#

By default, the Caching Module will pass the configured time-to-live (TTL) to the Caching Module Provider when caching data. The Caching Module Provider may also have its own default TTL. The cache isn't invalidated until the configured TTL passes.

Alternatively, you can set a custom TTL for a query. This is useful if you want the cached data to be invalidated sooner or later than the default TTL.

To set the TTL of the cached query results to a custom value, use the cache.ttl option:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    ttl: 100, // 100 seconds8  }9})

In the example above, you set the TTL of the cached query result to 100 seconds. It will be invalidated after that time.

You can also pass a function as the value of cache.ttl:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4  filters: {5    id: "prod_123"6  }7}, {8  cache: {9    enable: true,10    ttl: (args) => {11      return args[0].filters.id === "test" ? 10 : 10012    }13  }14})

In the example above, you use a function to determine the TTL. The function accepts the arguments passed to query.index as an array.

Then, you set the TTL based on the ID of the product passed in the filters.

Set Auto Invalidation#

By default, the Caching Module automatically invalidates cached query results when the data changes.

Alternatively, you can disable auto invalidation of cached query results. This is useful if you want to manage invalidating the cache manually.

To configure invalidation behavior, use the cache.autoInvalidate option:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    autoInvalidate: false,8  }9})

In this example, you disable auto invalidation of the query result. You must invalidate the cached data manually.

You can also pass a function as the value of cache.autoInvalidate:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    autoInvalidate: (args) => {8      return !args[0].fields.includes("custom_field")9    }10  }11})

In the example above, you use a function to determine whether to invalidate the cached query result automatically. The function accepts the arguments passed to query.index as an array.

Then, you enable auto-invalidation only if the fields passed to query.index don't include custom_fields. If this disables auto-invalidation, you must invalidate the cached data manually.

Set Caching Provider#

By default, the Caching Module uses the default Caching Module Provider to cache a query.

Alternatively, you can set the caching provider to use for a query. This is useful if you have multiple caching providers configured, and you want to use a specific one for a query, or you want to specify a fallback provider.

To configure the caching providers, use the cache.providers option:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4}, {5  cache: {6    enable: true,7    providers: ["caching-redis", "caching-memcached"]8  }9})

In the example above, you specify the providers with ID caching-redis and caching-memcached to cache the query results. These IDs must match the IDs of the providers in medusa-config.ts.

When you pass multiple providers, the cache is stored and retrieved in those providers in order.

You can also pass a function as the value of cache.providers:

Code
1const { data: products } = await query.index({2  entity: "product",3  fields: ["id", "title"],4  filters: {5    id: "prod_123"6  }7}, {8  cache: {9    enable: true,10    providers: (args) => {11      return args[0].filters.id === "test" ? ["caching-redis"] : ["caching-memcached"]12    }13  }14})

In the example above, you use a function to determine the caching providers. The function accepts the arguments passed to query.index as an array.

Then, you set the providers based on the ID of the product passed in the filters.

index Method Usage Examples#

The following sections show examples of how to use the index method in different scenarios.

Retrieve Linked Data Models#

Retrieve the records of a linked data model by passing in fields the data model's name suffixed with .*.

For example:

src/api/custom/products/route.ts
1const { data: products } = await query.index({2  entity: "product",3  fields: ["*", "brand.*"],4})

This will return all products with their linked brand data model.

Use Advanced Filters#

When setting filters on properties, you can use advanced filters like $ne and $gt. These are the same advanced filters accepted by the listing methods generated by the Service Factory.

For example, to only retrieve products linked to a brand:

src/api/custom/products/route.ts
1const { 2  data: products,3} = await query.index({4  entity: "product",5  fields: ["*", "brand.*"],6  filters: {7    brand: {8      id: {9        $ne: null,10      },11    },12  },13})

You use the $ne operator to filter products that are linked to a brand.

Another example is to retrieve products whose brand name starts with Acme:

src/api/custom/products/route.ts
1const { 2  data: products,3} = await query.index({4  entity: "product",5  fields: ["*", "brand.*"],6  filters: {7    brand: {8      name: {9        $like: "Acme%",10      },11    },12  },13})

This will return all products whose brand name starts with Acme.

Use Request Query Configurations#

API routes using the graph method can configure default query configurations, such as which fields to retrieve, while also allowing clients to override them using query parameters.

The index method supports the same configurations. For example, if you add the request query configuration as explained in the Query documentation, you can use those configurations in the index method:

src/api/custom/products/route.ts
1import {2  MedusaRequest,3  MedusaResponse,4} from "@medusajs/framework/http"5
6export const GET = async (7  req: MedusaRequest,8  res: MedusaResponse9) => {10  const query = req.scope.resolve("query")11  12  const { 13    data: products,14    metadata,15  } = await query.index({16    entity: "product",17    ...req.queryConfig,18    filters: {19      brand: {20        name: "Acme",21      },22    },23  })24
25  res.json({ products, ...metadata })26}

You pass the req.queryConfig object to the index method, which will contain the fields and pagination properties to use in the query.

Use Index Module in Workflows#

In a workflow's step, you can resolve query and use its index method to retrieve data using the Index Module.

For example:

src/workflows/custom-workflow.ts
1import {2  createStep,3  createWorkflow,4  StepResponse,5  WorkflowResponse,6} from "@medusajs/framework/workflows-sdk"7
8const retrieveBrandsStep = createStep(9  "retrieve-brands",10  async ({}, { container }) => {11    const query = container.resolve("query")12
13    const { data: brands } = await query.index({14      entity: "brand",15      fields: ["*", "products.*"],16      filters: {17        products: {18          id: {19            $ne: null,20          },21        },22      },23    })24
25    return new StepResponse(brands)26  }27)28
29export const retrieveBrandsWorkflow = createWorkflow(30  "retrieve-brands",31  () => {32    const retrieveBrands = retrieveBrandsStep()33
34    return new WorkflowResponse(retrieveBrands)35  }36)

This will retrieve all brands that are linked to at least one product.