
# Aggregations

Groupings, projections, and aggregation functions

Aggregations are a way to process the results of a search query. Aggregation allows you to group, sort, and transform your result data, and to extract analytic insights from it. Much like aggregation queries in other databases and search engines, they can be used to create analytics reports, or perform [faceted search](https://en.wikipedia.org/wiki/Faceted_search) style queries.

For example, indexing a web-server's logs, you can create a report for unique users by hour, country, or any other breakdown. Or you can create different reports for errors, warnings, etc.

## Core concepts

The basic idea of an aggregate query is this:

*   Perform a search query, filtering for records you wish to process.
*   Build a pipeline of operations that transform the results by zero or more sequences of:
    *   **Group and reduce**: grouping by fields in the results, and applying reducer functions to each group.
    *   **Sort**: sort the results based on one or more fields.
    *   **Apply transformations**: apply mathematical and string functions on fields in the pipeline, optionally creating new fields or replacing existing ones.
    *   **Limit**: limit the result, regardless of how the result is sorted.
    *   **Filter**: filter the results (post-query) based on predicates relating to its values.

The pipeline is dynamic and re-entrant, and every operation can be repeated. For example, you can group by property X, sort the top 100 results by group size, then group by property Y and sort the results by some other property, then apply a transformation on the output.

Figure 1: Aggregation Pipeline Example

![](../../img/pipeline.png)

## Aggregate request format

The aggregate request's syntax is defined as follows:

```sql
FT.AGGREGATE
  {index_name:string}
  {query_string:string}
  [VERBATIM]
  [LOAD {nargs:integer} {property:string} ...]
  [GROUPBY
    {nargs:integer} {property:string} ...
    REDUCE
      {FUNC:string}
      {nargs:integer} {arg:string} ...
      [AS {name:string}]
    ...
  ] ...
  [SORTBY
    {nargs:integer} {string} ...
    [MAX {num:integer}] ...
  ] ...
  [APPLY
    {EXPR:string}
    AS {name:string}
  ] ...
  [FILTER {EXPR:string}] ...
  [LIMIT {offset:integer} {num:integer} ] ...
  [PARAMS {nargs} {name} {value} ... ]
```

### Parameters in detail

Parameters that can take a variable number of arguments are expressed in the form of `param {nargs} {property_1... property_N}`. The first argument to the parameter is the number of arguments following the parameter. This allows Redis to avoid a parsing ambiguity in case one of your arguments has the name of another parameter. For example, to sort by first name, last name, and country, one would specify `SORTBY 6 firstName ASC lastName DESC country ASC`.

*   **index\_name**: The index the query is executed against.
    
*   **query\_string**: The base filtering query that retrieves the documents. It follows the exact same syntax as the search query, including filters, unions, not, optional, etc.
    
*   **LOAD {nargs} {property} ...** : Load document fields from the document HASH objects. This should be avoided as a general rule of thumb. Fields needed for aggregations should be stored as SORTABLE (and optionally UNF to avoid any normalization), where they are available to the aggregation pipeline with very low latency. LOAD hurts the performance of aggregate queries considerably since every processed record needs to execute the equivalent of HMGET against a Redis key, which when executed over millions of keys, amounts to very high processing times. The document ID can be loaded using `@__key`.
    
*   **GROUPBY {nargs} {property} ...** : Group the results in the pipeline based on one or more properties. Each group should have at least one reducer (See below), a function that handles the group entries, either counting them or performing multiple aggregate operations (see below).
    
*   **REDUCE {func} {nargs} {arg} ... \[AS {name}\]**: Reduce the matching results in each group into a single record, using a reduction function. For example, COUNT will count the number of records in the group. See the Reducers section below for more details on available reducers.
    
    The reducers can have their own property names using the `AS {name}` optional argument. If a name is not given, the resulting name will be the name of the reduce function and the group properties. For example, if a name is not given to COUNT\_DISTINCT by property `@foo`, the resulting name will be `count_distinct(@foo)`.
    
*   **SORTBY {nargs} {property} {ASC|DESC} \[MAX {num}\]**: Sort the pipeline up until the point of SORTBY, using a list of properties. By default, sorting is ascending, but `ASC` or `DESC` can be added for each property. `nargs` is the number of sorting parameters, including ASC and DESC. for example: `SORTBY 4 @foo ASC @bar DESC`.
    
    `MAX` is used to optimized sorting, by sorting only for the n-largest elements. Although it is not connected to `LIMIT`, you usually need just `SORTBY … MAX` for common queries.
    
*   **APPLY {expr} AS {name}**: Apply a one-to-one transformation on one or more properties, and either store the result as a new property down the pipeline, or replace any property using this transformation. `expr` is an expression that can be used to perform arithmetic operations on numeric properties, or functions that can be applied on properties depending on their types (see below), or any combination thereof. For example: `APPLY "sqrt(@foo)/log(@bar) + 5" AS baz` will evaluate this expression dynamically for each record in the pipeline and store the result as a new property called baz, that can be referenced by further APPLY / SORTBY / GROUPBY / REDUCE operations down the pipeline.
    
*   **LIMIT {offset} {num}**. Limit the number of results to return just `num` results starting at index `offset` (zero based). AS mentioned above, it is much more efficient to use `SORTBY … MAX` if you are interested in just limiting the output of a sort operation.
    
    However, limit can be used to limit results without sorting, or for paging the n-largest results as determined by `SORTBY MAX`. For example, getting results 50-100 of the top 100 results is most efficiently expressed as `SORTBY 1 @foo MAX 100 LIMIT 50 50`. Removing the MAX from SORTBY will result in the pipeline sorting all the records and then paging over results 50-100.
    
*   **FILTER {expr}**. Filter the results using predicate expressions relating to values in each result. The expressions are applied post-query and relate to the current state of the pipeline. See FILTER Expressions below for full details.
    
*   **PARAMS {nargs} {name} {value}**. Define one or more value parameters. Each parameter has a name and a value. Parameters can be referenced in the query string by a `$`, followed by the parameter name, e.g., `$user`, and each such reference in the search query to a parameter name is substituted by the corresponding parameter value. For example, with parameter definition `PARAMS 4 lon 29.69465 lat 34.95126`, the expression `@loc:[$lon $lat 10 km]` would be evaluated to `@loc:[29.69465 34.95126 10 km]`. Parameters cannot be referenced in the query string where concrete values are not allowed, such as in field names, e.g., `@loc`
    

## Example

A log of visits to a website might look like the following, each record of which has the following fields/properties:

*   **url** (text, sortable)
*   **timestamp** (numeric, sortable) - Unix timestamp of visit entry.
*   **country** (tag, sortable)
*   **user\_id** (text, sortable, not indexed)

### Example 1: unique users by hour, ordered chronologically.

The first step is to determine the index name and the filtering query. A filter query of `*` means "get all records":

```
FT.AGGREGATE myIndex "*"
```

Next, group the results by hour. The data contains visit times as unix timestamps in second resolution, so you'll need to extract the hour component of the timestamp. To do so, add an APPLY step that strips the sub-hour information from the timestamp and stores is as a new property, `hour`:

```
FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour
```

Next, group the results by hour and count the distinct user ids in each hour. This is done by a GROUPBY/REDUCE step:

```
FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour

  GROUPBY 1 @hour
  	REDUCE COUNT_DISTINCT 1 @user_id AS num_users
```

Next, sort the results by hour, ascending:

```
FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour

  GROUPBY 1 @hour
  	REDUCE COUNT_DISTINCT 1 @user_id AS num_users

  SORTBY 2 @hour ASC
```

And as a final step, format the hour as a human readable timestamp. This is done by calling the transformation function `timefmt` that formats Unix timestamps. You can specify a format to be passed to the system's `strftime` function ([see documentation](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html)), but not specifying one is equivalent to specifying `%FT%TZ` to `strftime`.

```
FT.AGGREGATE myIndex "*"
  APPLY "@timestamp - (@timestamp % 3600)" AS hour

  GROUPBY 1 @hour
  	REDUCE COUNT_DISTINCT 1 @user_id AS num_users

  SORTBY 2 @hour ASC

  APPLY timefmt(@hour) AS hour
```

### Example 2: Sort visits to a specific URL by day and country:

The next example filters by the url, transforms the timestamp to its day part, and groups by the day and country, counting the number of visits per group, sorting by day ascending and country descending.

```
FT.AGGREGATE myIndex "@url:\"about.html\""
    APPLY "@timestamp - (@timestamp % 86400)" AS day
    GROUPBY 2 @day @country
    	REDUCE count 0 AS num_visits
    SORTBY 4 @day ASC @country DESC
```

## GROUPBY reducers

`GROUPBY` works similarly to SQL `GROUP BY` clauses, and creates groups of results based on one or more properties in each record. For each group, Redis returns the group keys, or the values common to all records in the group, and the results of zero or more `REDUCE` clauses.

Each `GROUPBY` step in the pipeline may be accompanied by zero or more `REDUCE` clauses. Reducers apply an accumulation function to each record in the group and reduces them into a single record representing the group. When the processing is complete, all the records upstream of the `GROUPBY` step emit their reduced record.

For example, the simplest reducer is COUNT, which simply counts the number of records in each group.

If multiple `REDUCE` clauses exist for a single `GROUPBY` step, each reducer works independently on each result and writes its final output once. Each reducer may have its own alias determined using the `AS` optional parameter. If `AS` is not specified, the alias is the reduce function and its parameters, e.g. `count_distinct(foo,bar)`.

### Supported GROUPBY reducers

#### COUNT

**Format**

```
REDUCE COUNT 0
```

**Description**

Count the number of records in each group

#### COUNT\_DISTINCT

**Format**

```
REDUCE COUNT_DISTINCT 1 {property}
```

**Description**

Count the number of distinct values for `property`.

Note:

The reducer creates a hash-set per group, and hashes each record. This can be memory heavy if the groups are big.

#### COUNT\_DISTINCTISH

**Format**

```
REDUCE COUNT_DISTINCTISH 1 {property}
```

**Description**

Same as COUNT\_DISTINCT, provides an approximation instead of an exact count, which consumes less memory and CPU for big groups.

Note:

The reducer uses [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) counters per group, at ~3% error rate, and 1024 bytes of constant space per group. This means it is ideal for a few huge groups and not ideal for many small groups. In the former case, it can be an order of magnitude faster and consume much less memory than COUNT\_DISTINCT, but again, it does not fit every use case.

#### SUM

**Format**

```
REDUCE SUM 1 {property}
```

**Description**

Return the sum of all numeric values of a given property in a group. Non-numeric values in the group are counted as 0.

#### MIN

**Format**

```
REDUCE MIN 1 {property}
```

**Description**

Return the minimal value of a property, whether it is a string, number, or NULL.

#### MAX

**Format**

```
REDUCE MAX 1 {property}
```

**Description**

Return the maximal value of a property, whether it is a string, number or NULL.

#### AVG

**Format**

```
REDUCE AVG 1 {property}
```

**Description**

Return the average value of a numeric property. This is equivalent to reducing by sum and count, and later, applying the ratio of them as an APPLY step.

#### STDDEV

**Format**

```
REDUCE STDDEV 1 {property}
```

**Description**

Return the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation) of a numeric property in the group.

#### QUANTILE

**Format**

```
REDUCE QUANTILE 2 {property} {quantile}
```

**Description**

Return the value of a numeric property at a given quantile of the results. Quantile is expressed as a number between 0 and 1. For example, the median can be expressed as the quantile at 0.5, e.g. `REDUCE QUANTILE 2 @foo 0.5 AS median` .

If multiple quantiles are required, just repeat the QUANTILE reducer for each quantile. For example, `REDUCE QUANTILE 2 @foo 0.5 AS median REDUCE QUANTILE 2 @foo 0.99 AS p99`.

#### TOLIST

**Format**

```
REDUCE TOLIST 1 {property}
```

**Description**

Merge all distinct values of a given property into a single array.

#### FIRST\_VALUE

**Format**

```
REDUCE FIRST_VALUE {nargs} {property} [BY {property} [ASC|DESC]]
```

**Description**

Return the first or top value of a given property in the group, optionally by comparing it to another property. For example, you can extract the name of the oldest user in the group:

```
REDUCE FIRST_VALUE 4 @name BY @age DESC
```

If no `BY` is specified, the first value encountered in the group is returned.

If you wish to get the top or bottom value in the group sorted by the same value, you are better off using the `MIN/MAX` reducers, but the same effect will be achieved by doing `REDUCE FIRST_VALUE 4 @foo BY @foo DESC`.

#### RANDOM\_SAMPLE

**Format**

```
REDUCE RANDOM_SAMPLE {nargs} {property} {sample_size}
```

**Description**

Perform a reservoir sampling of the group elements with a given size, and return an array of the sampled items with an even distribution.

## APPLY expressions

`APPLY` performs a one-to-one transformation on one or more properties in each record. It either stores the result as a new property down the pipeline, or replaces any property using this transformation.

The transformations are expressed as a combination of arithmetic expressions and built in functions. Evaluating functions and expressions is recursively nested and can be composed without limit. For example: `sqrt(log(foo) * floor(@bar/baz)) + (3^@qaz % 6)` or simply `@foo/@bar`.

If an expression or a function is applied to values that do not match the expected types, no error is emitted and a NULL value is set as the result.

APPLY steps must have an explicit alias determined by the `AS` parameter.

### Literals inside expressions

*   Numbers are expressed as integers or floating point numbers, e.g., `2`, `3.141`, and `-34`. `inf` and `-inf` are acceptable as well.
*   Strings are quoted with either single or double quotes. Single quotes are acceptable inside strings quoted with double quotes and vice versa. Punctuation marks can be escaped with backslashes. e.g. `"foo's bar"` ,`'foo\'s bar'`, `"foo \"bar\""` .
*   Any literal or sub-expression can be wrapped in parentheses to resolve ambiguities of operator precedence.

### Arithmetic operations

For numeric expressions and properties, addition (`+`), subtraction (`-`), multiplication (`*`), division (`/`), modulo (`%`), and power (`^`) are supported. Bitwise logical operators are not supported.

Note that these operators apply only to numeric values and numeric sub-expressions. Any attempt to multiply a string by a number, for instance, will result in a NULL output.

### List of field APPLY functions

Function

Description

Example

exists(s)

Checks whether a field exists in a document.

`exists(@field)`

case(condition, if\_true, if\_false)

If condition is non-zero, return if\_true, otherwise return if\_false.

`case(exists(@foo), @foo, "no foo")`

### List of numeric APPLY functions

Function

Description

Example

log(x)

Return the logarithm of a number, property or subexpression

`log(@foo)`

abs(x)

Return the absolute number of a numeric expression

`abs(@foo-@bar)`

ceil(x)

Round to the smallest value not less than x

`ceil(@foo/3.14)`

floor(x)

Round to largest value not greater than x

`floor(@foo/3.14)`

log2(x)

Return the logarithm of x to base 2

`log2(2^@foo)`

exp(x)

Return the exponent of x, e.g., `e^x`

`exp(@foo)`

sqrt(x)

Return the square root of x

`sqrt(@foo)`

### List of string APPLY functions

Function

upper(s)

Return the uppercase conversion of s

`upper('hello world')`

lower(s)

Return the lowercase conversion of s

`lower("HELLO WORLD")`

startswith(s1,s2)

Return `1` if s2 is the prefix of s1, `0` otherwise.

`startswith(@field, "company")`

contains(s1,s2)

Return the number of occurrences of s2 in s1, `0` otherwise. If s2 is an empty string, return `length(s1) + 1`.

`contains(@field, "pa")`

strlen(s)

Return the length of s

`strlen(@t)`

substr(s, offset, count)

Return the substring of s, starting at _offset_ and having _count_ characters.  
If offset is negative, it represents the distance from the end of the string.  
If count is -1, it means "the rest of the string starting at offset".

`substr("hello", 0, 3)`  
`substr("hello", -2, -1)`

format( fmt, ...)

Use the arguments following `fmt` to format a string.  
Currently the only format argument supported is `%s` and it applies to all types of arguments.

`format("Hello, %s, you are %s years old", @name, @age)`

matched\_terms(\[max\_terms=100\])

Return the query terms that matched for each record (up to 100), as a list. If a limit is specified, Redis will return the first N matches found, based on query order.

`matched_terms()`

split(s, \[sep=","\], \[strip=" "\])

Split a string by any character in the string sep, and strip any characters in strip. If only s is specified, it is split by commas and spaces are stripped. The output is an array.

split("foo,bar")

### List of date/time APPLY functions

Function

Description

timefmt(x, \[fmt\])

Return a formatted time string based on a numeric timestamp value x.  
See [strftime](https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html) for formatting options.  
Not specifying `fmt` is equivalent to `%FT%TZ`.

parsetime(timesharing, \[fmt\])

The opposite of timefmt() - parse a time format using a given format string

day(timestamp)

Round a Unix timestamp to midnight (00:00) start of the current day.

hour(timestamp)

Round a Unix timestamp to the beginning of the current hour.

minute(timestamp)

Round a Unix timestamp to the beginning of the current minute.

month(timestamp)

Round a unix timestamp to the beginning of the current month.

dayofweek(timestamp)

Convert a Unix timestamp to the day number (Sunday = 0).

dayofmonth(timestamp)

Convert a Unix timestamp to the day of month number (1 .. 31).

dayofyear(timestamp)

Convert a Unix timestamp to the day of year number (0 .. 365).

year(timestamp)

Convert a Unix timestamp to the current year (e.g. 2018).

monthofyear(timestamp)

Convert a Unix timestamp to the current month (0 .. 11).

### List of geo APPLY functions

Function

Description

Example

geodistance(field,field)

Return distance in meters.

`geodistance(@field1,@field2)`

geodistance(field,"lon,lat")

Return distance in meters.

`geodistance(@field,"1.2,-3.4")`

geodistance(field,lon,lat)

Return distance in meters.

`geodistance(@field,1.2,-3.4)`

geodistance("lon,lat",field)

Return distance in meters.

`geodistance("1.2,-3.4",@field)`

geodistance("lon,lat","lon,lat")

Return distance in meters.

`geodistance("1.2,-3.4","5.6,-7.8")`

geodistance("lon,lat",lon,lat)

Return distance in meters.

`geodistance("1.2,-3.4",5.6,-7.8)`

geodistance(lon,lat,field)

Return distance in meters.

`geodistance(1.2,-3.4,@field)`

geodistance(lon,lat,"lon,lat")

Return distance in meters.

`geodistance(1.2,-3.4,"5.6,-7.8")`

geodistance(lon,lat,lon,lat)

Return distance in meters.

`geodistance(1.2,-3.4,5.6,-7.8)`

```
FT.AGGREGATE myIdx "*"  LOAD 1 location  APPLY "geodistance(@location,\"-1.1,2.2\")" AS dist
```

To retrieve the distance:

```
FT.AGGREGATE myIdx "*"  LOAD 1 location  APPLY "geodistance(@location,\"-1.1,2.2\")" AS dist
```

Note: the geo field must be preloaded using `LOAD`.

Results can also be sorted by distance:

```
FT.AGGREGATE idx "*" LOAD 1 @location FILTER "exists(@location)" APPLY "geodistance(@location,-117.824722,33.68590)" AS dist SORTBY 2 @dist DESC
```

Note: Make sure no location is missing, otherwise the SORTBY will not return any results. Use FILTER to make sure you do the sorting on all valid locations.

## FILTER expressions

FILTER expressions filter the results using predicates relating to values in the result set.

The FILTER expressions are evaluated post-query and relate to the current state of the pipeline. Thus they can be useful to prune the results based on group calculations. Note that the filters are not indexed and will not speed up processing.

Filter expressions follow the syntax of APPLY expressions, with the addition of the conditions `==`, `!=`, `<`, `<=`, `>`, `>=`. Two or more predicates can be combined with logical AND (`&&`) and OR (`||`). A single predicate can be negated with a NOT prefix (`!`).

For example, filtering all results where the user name is 'foo' and the age is less than 20 is expressed as:

```
FT.AGGREGATE
  ...
  FILTER "@name=='foo' && @age < 20"
  ...
```

Several filter steps can be added, although at the same stage in the pipeline, it is more efficient to combine several predicates into a single filter step.

## Cursor API

```
FT.AGGREGATE ... WITHCURSOR [COUNT {read size} MAXIDLE {idle timeout}]
FT.CURSOR READ {idx} {cid} [COUNT {read size}]
FT.CURSOR DEL {idx} {cid}
```

You can use cursors with [`FT.AGGREGATE`](/docs/latest/commands/ft.aggregate/), with the `WITHCURSOR` keyword. Cursors allow you to consume only part of the response, allowing you to fetch additional results as needed. This is much quicker than using `LIMIT` with offset, since the query is executed only once, and its state is stored on the server.

To use cursors, specify the `WITHCURSOR` keyword in [`FT.AGGREGATE`](/docs/latest/commands/ft.aggregate/). For example:

```
FT.AGGREGATE idx * WITHCURSOR
```

This will return a response of an array with two elements. The first element is the actual (partial) result, and the second is the cursor ID. The cursor ID can then be fed to [`FT.CURSOR READ`](/docs/latest/commands/ft.cursor-read/) repeatedly until the cursor ID is 0, in which case all results have been returned.

To read from an existing cursor, use [`FT.CURSOR READ`](/docs/latest/commands/ft.cursor-read/). For example:

```
FT.CURSOR READ idx 342459320
```

Assuming `342459320` is the cursor ID returned from the [`FT.AGGREGATE`](/docs/latest/commands/ft.aggregate/) request, here is an example in pseudo-code:

```
response, cursor = FT.AGGREGATE "idx" "redis" "WITHCURSOR";
while (1) {
  processResponse(response)
  if (!cursor) {
    break;
  }
  response, cursor = FT.CURSOR read "idx" cursor
}
```

Note that even if the cursor is 0, a partial result may still be returned.

### Cursor settings

#### Read size

You can control how many rows are read for each cursor fetch by using the `COUNT` parameter. This parameter can be specified both in [`FT.AGGREGATE`](/docs/latest/commands/ft.aggregate/) (immediately after `WITHCURSOR`) or in [`FT.CURSOR READ`](/docs/latest/commands/ft.cursor-read/).

The following example will read 10 rows at a time:

```
FT.AGGREGATE idx query WITHCURSOR COUNT 10
```

You can override this setting by also specifying `COUNT` in `CURSOR READ`. The following example will return at most 50 results:

```
FT.CURSOR READ idx 342459320 COUNT 50
```

The default read size is 1000.

#### Timeouts and limits

Because cursors are stateful resources that occupy memory on the server, they have a limited lifetime. To safeguard against orphaned/stale cursors, cursors have an idle timeout value. If no activity occurs on the cursor before the idle timeout, the cursor is deleted. The idle timer resets to 0 whenever the cursor is read from using `CURSOR READ`.

The default idle timeout is 300000 milliseconds (or 300 seconds). You can modify the idle timeout using the `MAXIDLE` keyword when creating the cursor. Note that the value cannot exceed the default 300s.

For example, to set a limit of ten seconds:

```
FT.AGGREGATE idx query WITHCURSOR MAXIDLE 10000
```

During a cursor’s lifetime, CURSOR READ may return results that exclude newer updates.

Also, note that in case of any load rebalancing operations (such as Atomic Slot Migration) occurring during the lifetime of a cursor, some results may be missing. In a Redis Software deployment, these operations may happen behind the scenes without any user intervention.

### Other cursor commands

Cursors can be explicitly deleted using the `CURSOR DEL` command. For example:

```
FT.CURSOR DEL idx 342459320
```

Note that cursors are automatically deleted if all their results have been returned, or if they have timed out.

All idle cursors can be forcefully purged at the same time using `FT.CURSOR GC idx 0` command. By default, Redis uses a lazy throttled approach to garbage collection, which collects idle cursors every 500 operations, or every second, whichever is later.

## On this page
