Skip to content

Add cursor-based pagination with Page Token implementation #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Add cursor-based pagination with Page Token implementation #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6138eda
Initial plan
Copilot Jan 28, 2026
a3eb233
Add cursor-based pagination infrastructure with HasCursorKey builder …
Copilot Jan 28, 2026
417c3ff
Add cursor pagination demo and comprehensive tests
Copilot Jan 28, 2026
26f6f57
Add cursor pagination documentation to README
Copilot Jan 28, 2026
273c935
Address code review feedback: improve error handling, add type suppor…
Copilot Jan 28, 2026
4e0c18b
Reorganize cursor pagination tests to follow project naming conventions
Copilot Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 111 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@
- **Dynamic Query Building**: Generate queries dynamically using expression trees.
- **Filtering**: Apply flexible filtering logic to refine query results.
- **Field Projection**: Return only the specified fields to optimize API responses.
- **Pagination and Sorting**: Built-in support for pagination and sorting.
- **Pagination and Sorting**: Built-in support for both offset-based and cursor-based pagination, plus sorting.
- **Cursor-Based Pagination**: Efficient pagination using opaque page tokens for large datasets.
- **ASP.NET Core Integration**: Middleware support for easy integration into ASP.NET Core projects.

## Benchmark
Expand Down Expand Up @@ -192,6 +193,115 @@ GET /Users?filter[ids]=1&filter[ids]=3&fields=Id,Name&sort=-Id&page=1&pageSize=2
}
```

## Cursor-Based Pagination

AutoQuery also supports cursor-based pagination, which is more efficient for large datasets and provides consistent results even when data is being modified. Unlike offset-based pagination, cursor-based pagination uses opaque page tokens to track position.

### Benefits of Cursor-Based Pagination

- **Consistent Results**: No duplicate or skipped items when data changes between requests
- **Better Performance**: More efficient for large datasets as it doesn't require counting or skipping rows
- **Scalability**: Works well with real-time data and high-volume scenarios

### Using Cursor-Based Pagination

1. Define a query options class implementing `IQueryCursorOptions`:
```csharp
public class UserCursorQueryOptions : IQueryCursorOptions
{
[FromQuery(Name = "filter[name]")]
public string? FilterName { get; set; }
[FromQuery(Name = "fields")]
public string? Fields { get; set; }
[FromQuery(Name = "sort")]
public string? Sort { get; set; }
[FromQuery(Name = "pageToken")]
public string? PageToken { get; set; }
[FromQuery(Name = "pageSize")]
public int? PageSize { get; set; }
}
```

2. Configure the cursor key in your filter configuration:
```csharp
public class UserCursorQueryConfiguration : IFilterQueryConfiguration<UserCursorQueryOptions, User>
{
public void Configure(FilterQueryBuilder<UserCursorQueryOptions, User> builder)
{
// Configure cursor key for pagination
builder.HasCursorKey(d => d.Id);

// Configure filter properties
builder.Property(q => q.FilterName, d => d.Name)
.HasEqual();
}
}
```

3. Create a controller endpoint using cursor pagination:
```csharp
[HttpGet("cursor")]
[EnableFieldProjection]
public IActionResult GetWithCursor(UserCursorQueryOptions queryOptions)
{
var result = users.AsQueryable()
.ApplyQueryCursorPaged(_queryProcessor, queryOptions);
return Ok(result);
}
```

4. Example Request (First Page):
```http
GET /Users/cursor?pageSize=2&sort=id
```

5. Example Response (First Page):
```json
{
"datas": [
{
"id": 1,
"name": "John Doe",
"email": "john.doe@example.com"
},
{
"id": 2,
"name": "Jane Smith",
"email": "jane.smith@example.com"
}
],
"nextPageToken": "Mg==",
"count": 2
}
```

6. Example Request (Next Page):
```http
GET /Users/cursor?pageSize=2&sort=id&pageToken=Mg==
```

7. Example Response (Next Page):
```json
{
"datas": [
{
"id": 3,
"name": "Alice Johnson",
"email": "alice.johnson@example.com"
},
{
"id": 4,
"name": "Bob Brown",
"email": "bob.brown@example.com"
}
],
"nextPageToken": "NA==",
"count": 2
}
```

**Note**: When `nextPageToken` is `null`, there are no more results to fetch.

## Contribution

Contributions are welcome! Feel free to submit issues or pull requests to improve the project.
Expand Down
21 changes: 21 additions & 0 deletions sample/AutoQueryApiDemo/Configurations/UserCursorQueryConfiguration.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
using AutoQuery;
using AutoQuery.Abstractions;
using AutoQuery.Extensions;
using AutoQueryApiDemo.Models;

namespace AutoQueryApiDemo.Configurations;

public class UserCursorQueryConfiguration : IFilterQueryConfiguration<UserCursorQueryOptions, User>
{
public void Configure(FilterQueryBuilder<UserCursorQueryOptions, User> builder)
{
// Configure cursor key for cursor-based pagination
builder.HasCursorKey(d => d.Id);

// Configure filter properties
builder.Property(q => q.FilterIds, d => d.Id)
.HasCollectionContains();
builder.Property(q => q.FilterName, d => d.Name)
.HasEqual();
}
}
9 changes: 9 additions & 0 deletions sample/AutoQueryApiDemo/Controllers/UsersController.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,13 @@ public IActionResult Get(UserQueryOptions queryOptions)
.ApplyQueryPagedResult(_queryProcessor, queryOptions);
return Ok(result);
}

[HttpGet("cursor")]
[EnableFieldProjection]
public IActionResult GetWithCursor(UserCursorQueryOptions queryOptions)
{
var result = users.AsQueryable()
.ApplyQueryCursorPaged(_queryProcessor, queryOptions);
return Ok(result);
}
}
20 changes: 20 additions & 0 deletions sample/AutoQueryApiDemo/Models/UserCursorQueryOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
using AutoQuery.Abstractions;
using Microsoft.AspNetCore.Mvc;

namespace AutoQueryApiDemo.Models;

public class UserCursorQueryOptions : IQueryCursorOptions
{
[FromQuery(Name = "filter[ids]")]
public int[]? FilterIds { get; set; }
[FromQuery(Name = "filter[name]")]
public string? FilterName { get; set; }
[FromQuery(Name = "fields")]
public string? Fields { get; set; }
[FromQuery(Name = "sort")]
public string? Sort { get; set; }
[FromQuery(Name = "pageToken")]
public string? PageToken { get; set; }
[FromQuery(Name = "pageSize")]
public int? PageSize { get; set; }
}
17 changes: 17 additions & 0 deletions src/AutoQuery/Abstractions/IQueryCursorOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
namespace AutoQuery.Abstractions;

/// <summary>
/// Query parameters for cursor-based pagination.
/// </summary>
public interface IQueryCursorOptions : IQueryOptions
{
/// <summary>
/// Page token representing the cursor position for pagination.
/// </summary>
string? PageToken { get; set; }

/// <summary>
/// Number of items to return.
/// </summary>
int? PageSize { get; set; }
}
9 changes: 8 additions & 1 deletion src/AutoQuery/Abstractions/IQueryProcessor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,12 @@ public interface IQueryProcessor
/// <returns>The selector expression, or null if no selection conditions exist.</returns>
Expression<Func<TData, TData>>? BuildSelectorExpression<TData, TQueryOptions>(TQueryOptions queryOptions)
where TQueryOptions : IQueryOptions;
}

/// <summary>
/// Gets the cursor key selector for cursor-based pagination.
/// </summary>
/// <typeparam name="TQueryOptions">The type of the query options.</typeparam>
/// <typeparam name="TData">The type of the data.</typeparam>
/// <returns>The cursor key selector expression, or null if not configured.</returns>
LambdaExpression? GetCursorKeySelector<TQueryOptions, TData>();
}
10 changes: 10 additions & 0 deletions src/AutoQuery/CursorPagedResult.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
namespace AutoQuery;

/// <summary>
/// Represents a cursor-based paginated result.
/// </summary>
/// <typeparam name="TData">The type of data contained in the result set.</typeparam>
/// <param name="Datas">The data collection of the paginated result, represented as <see cref="IEnumerable{T}"/>.</param>
/// <param name="NextPageToken">The page token for the next page, or null if there are no more results.</param>
/// <param name="Count">The number of items in the current result set.</param>
public record CursorPagedResult<TData>(IEnumerable<TData> Datas, string? NextPageToken, int Count);
98 changes: 98 additions & 0 deletions src/AutoQuery/Extensions/QueryExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,4 +158,102 @@ public static IQueryable<T> ApplyPaging<T>(this IQueryable<T> query, IQueryPaged

return query;
}

/// <summary>
/// Applies query conditions and cursor-based pagination.
/// </summary>
/// <typeparam name="TData">The type of the entity being queried.</typeparam>
/// <typeparam name="TQueryOptions">The type of the query options.</typeparam>
/// <param name="query">The query object.</param>
/// <param name="queryProcessor">The query processor.</param>
/// <param name="queryOption">The query options.</param>
/// <returns>The cursor-based paginated result.</returns>
public static CursorPagedResult<TData> ApplyQueryCursorPaged<TData, TQueryOptions>(
this IQueryable<TData> query,
IQueryProcessor queryProcessor,
TQueryOptions queryOption)
where TQueryOptions : IQueryCursorOptions
where TData : class
{
var filterExpression = queryProcessor.BuildFilterExpression<TData, TQueryOptions>(queryOption);
var selectorExpression = queryProcessor.BuildSelectorExpression<TData, TQueryOptions>(queryOption);

if (filterExpression != null)
query = query.Where(filterExpression);

if (selectorExpression != null)
query = query.Select(selectorExpression);

// Get cursor key selector from the query processor
var cursorKeySelector = queryProcessor.GetCursorKeySelector<TQueryOptions, TData>();

if (cursorKeySelector == null)
throw new InvalidOperationException($"Cursor key selector not configured for {typeof(TData).Name}. Use HasCursorKey() in your configuration.");

// Apply sorting - cursor pagination requires consistent ordering
query = query.ApplySort(queryOption);

// Apply cursor-based filtering if page token is provided
if (!string.IsNullOrWhiteSpace(queryOption.PageToken))
{
query = ApplyCursorFilter(query, cursorKeySelector, queryOption.PageToken);
}

// Fetch one extra item to determine if there are more results
var pageSize = queryOption.PageSize ?? 10;
var items = query.Take(pageSize + 1).ToList();

// Determine if there are more results and generate next page token
string? nextPageToken = null;
var hasMore = items.Count > pageSize;

if (hasMore)
{
items = items.Take(pageSize).ToList();
var lastItem = items.Last();
var cursorValue = GetCursorValue(lastItem, cursorKeySelector);
if (cursorValue == null)
throw new InvalidOperationException($"Cursor key value cannot be null for {typeof(TData).Name}.");
nextPageToken = PageToken.Encode(cursorValue);
}

return new CursorPagedResult<TData>(items, nextPageToken, items.Count);
}

/// <summary>
/// Applies cursor-based filtering to the query.
/// </summary>
private static IQueryable<TData> ApplyCursorFilter<TData>(
IQueryable<TData> query,
LambdaExpression cursorKeySelector,
string pageToken)
{
var returnType = ((cursorKeySelector.Body as MemberExpression)?.Type)
?? ((cursorKeySelector.Body as UnaryExpression)?.Operand as MemberExpression)?.Type
?? typeof(object);

var decodeMethod = typeof(PageToken).GetMethod(nameof(PageToken.Decode))!.MakeGenericMethod(returnType);
var cursorValue = decodeMethod.Invoke(null, new object[] { pageToken });

if (cursorValue == null)
throw new InvalidOperationException("Decoded cursor value cannot be null.");

// Build the filter expression: entity => entity.CursorKey > cursorValue
var parameter = Expression.Parameter(typeof(TData), "entity");
var cursorProperty = Expression.Invoke(cursorKeySelector, parameter);
var constant = Expression.Constant(cursorValue, returnType);
var greaterThan = Expression.GreaterThan(cursorProperty, constant);
var lambda = Expression.Lambda<Func<TData, bool>>(greaterThan, parameter);

return query.Where(lambda);
}

/// <summary>
/// Gets the cursor value from an entity.
/// </summary>
private static object? GetCursorValue<TData>(TData entity, LambdaExpression cursorKeySelector)
{
var compiled = cursorKeySelector.Compile();
return compiled.DynamicInvoke(entity);
}
}
22 changes: 22 additions & 0 deletions src/AutoQuery/FilterQueryBuilder.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ public class FilterQueryBuilder<TQueryOptions, TData>
private readonly ConcurrentDictionary<(Type BuilderType, Type QueryPropertyType), Func<object, object, Expression<Func<TData, bool>>>> _compiledExpressionsCache = new();
private readonly ConcurrentDictionary<PropertyInfo, Func<TQueryOptions, object>> _propertyAccessorsCache = new();
private readonly Dictionary<string, PropertyInfo> _queryOptionsProperties = typeof(TQueryOptions).GetProperties().ToDictionary(p => p.Name);
private LambdaExpression? _cursorKeySelector;

/// <summary>
/// Registers a property for use in filter queries.
Expand Down Expand Up @@ -159,4 +160,25 @@ private static Expression<Func<TData, bool>> CombineExpressions(Expression<Func<
);
return Expression.Lambda<Func<TData, bool>>(body, parameter);
}

/// <summary>
/// Configures the cursor key selector for cursor-based pagination.
/// </summary>
/// <typeparam name="TCursorKey">The type of the cursor key.</typeparam>
/// <param name="cursorKeySelector">The cursor key selector expression.</param>
/// <returns>The filter query builder for fluent chaining.</returns>
public FilterQueryBuilder<TQueryOptions, TData> HasCursorKey<TCursorKey>(Expression<Func<TData, TCursorKey>> cursorKeySelector)
{
_cursorKeySelector = cursorKeySelector ?? throw new ArgumentNullException(nameof(cursorKeySelector));
return this;
}

/// <summary>
/// Gets the cursor key selector.
/// </summary>
/// <returns>The cursor key selector expression, or null if not configured.</returns>
public LambdaExpression? GetCursorKeySelector()
{
return _cursorKeySelector;
}
}
Loading