Appearance
Search & Navigation
Overview
Search and navigation enable users to find, filter, and access content efficiently across the Progressive World-Building Platform. As projects grow -- accumulating dozens of timeline updates, characters, factions, and relationships -- the ability to locate specific content quickly becomes critical. A Game Master preparing for a session needs to find every mention of a particular NPC. A writer needs to locate all factions involved in a storyline. A player needs to revisit a character they saw weeks ago.
The search system is built on PostgreSQL's native full-text search capabilities (tsvector/tsquery), which provides relevance-ranked results without introducing external infrastructure. Search queries respect the same visibility rules that govern the rest of the platform: a Player searching for "vampire" will never see results from unpublished timeline updates or private characters. The system indexes content from all major entity types -- timeline updates, sections, items, characters, factions, and relationships -- and returns a unified result set with content type indicators and keyword highlighting.
Beyond search, this feature covers navigation aids that help users move through the platform efficiently: bookmarks for saving frequently accessed content, recently viewed item tracking, and consistent pagination patterns. Later phases introduce advanced filtering, cross-project search, data export, bulk operations, and content templates.
Goals
- Provide fast, relevant full-text search within a project using PostgreSQL tsvector/tsquery
- Index all major content types into a unified, searchable corpus
- Enforce permission-aware results so users only see content they are authorized to view
- Return highlighted keyword matches for context in search results
- Support filtering by content type, development status, creator, and date range (Phase 4)
- Enable cross-project search that respects project membership (Phase 4)
- Provide async project data export in JSON and PDF formats (Phase 4)
- Support bulk status changes for timeline updates with transactional safety (Phase 4)
- Offer pre-built and user-created templates for rapid content creation (Phase 4)
- Track user bookmarks and recently viewed items for efficient navigation
User Stories
Game Master / Storyteller
As a GM, I want to search my project for "vampire" so I can find every character, faction, and timeline item related to vampires before a session.
As a GM, I want to filter search results by content type so I can quickly narrow down to just characters or just factions.
As a GM, I want to export my entire project as JSON so I have a backup and can migrate data if needed.
As a GM, I want to bulk-publish all "in development" timeline updates after a session so I do not have to update them one at a time.
As a GM, I want to create a new character from a "D&D Character" template so I get pre-filled metadata fields (class, race, level) instead of starting from scratch.
Writer
As a writer, I want to search across all my projects for a character name so I can track where I have used them across different stories.
As a writer, I want to bookmark key characters and factions so I can quickly navigate to them while writing.
As a writer, I want to see my recently viewed items so I can pick up where I left off.
As a writer, I want to save one of my well-structured factions as a template so I can reuse that structure in future projects.
Player
As a player, I want to search for a character I met in-game so I can review their published details.
As a player, I want search results to show me only content I have access to so I am not confused by missing or restricted results.
As a player, I want to export the published content from a project so I can reference it offline.
Functional Description
Search (Phase 1 -- Basic)
Full-Text Search Index
The search system maintains a PostgreSQL full-text search index that spans all searchable content types within a project. Each searchable entity contributes specific fields to the index.
Indexed fields by content type:
| Content Type | Indexed Fields | Weight |
|---|---|---|
| TimelineUpdate | Title | A (highest) |
| Section | Label | B |
| Item | Name, Description | B, C |
| Character | Name, Description | A, C |
| Faction | Name, Description | A, C |
| Relationship | Description | C |
| FactionRelationship | Description | C |
PostgreSQL tsvector supports four weight classes (A, B, C, D) that influence relevance ranking. Names and titles carry the highest weight so that exact name matches rank above incidental description mentions.
Search index table:
A dedicated SearchIndex table stores pre-computed tsvector data for each indexed entity. This avoids computing tsvectors at query time, which would require joining across multiple tables.
| Field | Type | Notes |
|---|---|---|
| Id | Guid | Primary key |
| ProjectId | Guid | FK to Project, enables project-scoped queries |
| EntityId | Guid | FK to the source entity |
| EntityType | enum | timeline_update, section, item, character, faction, relationship, faction_relationship |
| ContentVector | tsvector | Pre-computed weighted search vector |
| DisplayTitle | string | Title or name shown in search results |
| DisplaySnippet | string | Short preview text for the result |
| CreatedBy | Guid | For filtering by creator |
| CreatedAt | DateTime | For date range filtering |
| UpdatedAt | DateTime | Tracks when the index row was last refreshed |
A GIN index on ContentVector enables fast full-text lookups. A composite index on (ProjectId, ContentVector) scopes queries to a single project efficiently.
Index maintenance:
Search index entries are updated whenever the source entity is created, modified, or deleted. This is driven by domain events:
TimelineUpdateCreatedEvent/TimelineUpdateUpdatedEvent-- upsert the index row for the timeline updateCharacterCreatedEvent/CharacterUpdatedEvent-- upsert the index row for the character- Similarly for sections, items, factions, and relationships
- Delete events remove the corresponding index row
Index updates are performed within the same database transaction as the source entity change, ensuring the index is always consistent with the data.
Search Query Pipeline
When a user submits a search query, the system processes it through a pipeline that enforces permissions, queries the index, and formats results.
Query parsing:
The raw query string is sanitized to remove PostgreSQL special characters and then converted into a tsquery. Multi-word queries are joined with the & (AND) operator by default, so searching for "vampire lord" finds content containing both words. Individual terms are run through the same language stemmer used during indexing (English by default) so that "running" matches "run".
Permission-aware filtering:
After the index query returns raw matches, the system applies permission filtering. This requires knowing:
- The user's role in the project (from
IPermissionService) - The visibility status of each matched entity
Filtering rules by entity type:
| Entity Type | Player/Viewer Filtering |
|---|---|
| TimelineUpdate | Exclude Concept and In Development status |
| Section | Exclude if parent TimelineUpdate is Concept or In Development |
| Item | Exclude if parent TimelineUpdate is Concept or In Development |
| Character | Exclude private characters (unless user is creator) |
| Faction | No additional filtering (factions have no visibility flag) |
| Relationship | Exclude private relationships (unless user is creator) |
| FactionRelationship | Exclude secret faction relationships |
For Owners, Storytellers, and Co-Creators, no filtering is applied -- they see all results.
Permission filtering is performed as part of the database query (using joins and WHERE clauses) rather than in application memory. This prevents loading large result sets only to discard most of them.
Result ranking:
PostgreSQL's ts_rank_cd function scores each result based on how well it matches the query, factoring in the weight classes assigned during indexing. Results are sorted by this relevance score in descending order.
Keyword highlighting:
The ts_headline function generates highlighted snippets showing the matched keywords in context. This provides users with immediate visual confirmation of why a result matched.
Unified Search Result Format
All search results share a common shape regardless of content type:
| Field | Type | Notes |
|---|---|---|
| entityType | string | "timeline_update", "character", "faction", etc. |
| publicId | string | Project-scoped public ID (e.g., "VNO-42") for aggregate roots, or parent public ID for child entities |
| title | string | Name or title of the matched entity |
| snippet | string | Highlighted text snippet showing match context |
| relevanceScore | float | Ranking score for sort order |
| parentContext | string | Human-readable breadcrumb (e.g., "Session 5 > NPCs Introduced") |
| updatedAt | DateTime | When the entity was last modified |
Child entities (sections, items, relationships) include a parentContext field so the user can understand where the result lives in the hierarchy. For example, an item result might show parentContext as "Session 5: The Blood Moon > New Characters".
Search Index Architecture
Search (Phase 4 -- Advanced)
Filtered Search
Advanced search extends the basic search endpoint with query parameters for narrowing results:
| Parameter | Type | Example | Behavior |
|---|---|---|---|
q | string | vampire | Full-text keyword query (required) |
type | string | character | Filter to a specific content type |
status | string | published | Filter by development status (timeline updates only) |
creator | Guid | user ID | Filter to content created by a specific user |
from | date | 2026-01-01 | Include only content created on or after this date |
to | date | 2026-06-01 | Include only content created on or before this date |
All filters are additive (AND logic). When type is specified, only results of that content type are returned. When status is specified, it applies only to entity types that have a status field (timeline updates); other entity types are unaffected.
Faceted Results
When no type filter is specified, the response includes facet counts showing how many results matched in each content type. This lets the UI display tabs or badges like "Characters (5), Factions (2), Timeline Updates (8)".
Facet counts are computed by grouping the permission-filtered results by EntityType and counting. This is performed in a single query alongside the main search using PostgreSQL's COUNT(*) ... GROUP BY on the filtered result set.
Cross-Project Search
The cross-project search endpoint (GET /api/search?q=keyword) searches across all projects where the requesting user is an active member.
Cross-project search results include an additional projectKey field so users can identify which project each result belongs to. Results are ranked by relevance score across all projects, not grouped by project.
Performance consideration: Cross-project search queries the SearchIndex table with a WHERE ProjectId IN (...) clause rather than issuing separate queries per project. Permission filtering is applied per-project based on the user's role in each project, using a single query with conditional visibility logic.
Meilisearch Migration Path
The search system is designed so that a future migration from PostgreSQL full-text search to Meilisearch (or another search engine) requires changes in only two places:
- Index maintenance -- Instead of writing to the
SearchIndexPostgreSQL table, the event handlers would push documents to the Meilisearch index via its API - Query execution -- Instead of building tsquery and querying PostgreSQL, the search service would call the Meilisearch search API
The search service interface, permission filtering logic, result format, and API contract remain unchanged. This is achieved by encapsulating all search engine interaction behind a ISearchEngine interface with two methods: IndexDocument and Search. The PostgreSQL implementation is the Phase 1 default; a Meilisearch implementation can be swapped in without touching controllers, permission logic, or result formatting.
Export (Phase 4)
Export Overview
Project export generates a downloadable file containing all project data visible to the requesting user. Exports are asynchronous because generating a complete project archive (especially PDF) can take significant time for large projects.
Export Formats
| Format | Contents | Use Case |
|---|---|---|
| JSON | Structured data: project metadata, timelines, characters, factions, relationships | Backup, data portability, programmatic access |
| Formatted document with table of contents, sections per content type | Offline reference, printing, sharing with non-platform users |
Export Job Lifecycle
Export request (POST /api/projects/{key}/export):
The request body specifies the desired format (json or pdf). The endpoint validates that the user is an active member of the project, creates an export job record, and returns a job ID immediately. The actual file generation happens asynchronously.
Export job record:
| Field | Type | Notes |
|---|---|---|
| Id | Guid | Export job identifier |
| ProjectId | Guid | FK to Project |
| RequestedBy | Guid | User who requested the export |
| Format | enum | json, pdf |
| Status | enum | requested, processing, ready, failed, expired |
| FilePath | string | Path to generated file (set when ready) |
| FileSize | long | Size in bytes (set when ready) |
| ErrorMessage | string | Error details (set when failed) |
| RequestedAt | DateTime | When the export was requested |
| CompletedAt | DateTime | When file generation finished |
| ExpiresAt | DateTime | When the file will be auto-deleted (RequestedAt + 24 hours) |
Export status check / download (GET /api/projects/{key}/export/{id}):
If the job is still processing, returns 202 Accepted with the current status. If the job is ready, returns the file as a download. If the job has failed, returns the error details. If the job has expired, returns 410 Gone.
Permission-filtered export:
The export contains only data the requesting user can see. A Player's export will not contain unpublished timeline updates, private characters, or private relationships. The export generation logic calls the same IPermissionService.FilterByVisibility method used by the regular API endpoints.
Export content structure (JSON):
The JSON export is a single file containing the complete project graph:
- Project metadata (name, description, key, theme)
- Timeline updates with their sections, items, and sub-items
- Characters with their metadata
- Factions with memberships
- Character relationships
- Faction relationships
File storage and cleanup:
Generated export files are stored in temporary server-side storage (local filesystem or cloud blob storage, depending on deployment). A background cleanup job runs periodically (every hour) and deletes files whose ExpiresAt has passed. The corresponding export job record is updated to expired status.
Bulk Operations (Phase 4)
Bulk Status Change
The bulk status change endpoint (POST /api/projects/{key}/timelines/bulk-status) allows creators to change the development status of multiple timeline updates in a single operation.
Request body:
| Field | Type | Notes |
|---|---|---|
| timelineSequenceNumbers | int[] | List of timeline update sequence numbers to modify |
| targetStatus | enum | The status to transition to (published, in-development, concept, legacy) |
Validation rules:
- All sequence numbers must exist within the project
- Every status transition must be valid per the timeline status state machine (see Content Timeline System)
- The user must have
CanEditTimelinepermission for the project
Transactional behavior:
Bulk status changes are all-or-nothing. If any single transition is invalid, the entire operation is rejected and no changes are persisted. The response includes details about which specific transitions failed and why, so the user can correct the request.
Domain events:
Each successful status change within the bulk operation raises a TimelineStatusChangedEvent. These events trigger the same downstream effects as individual status changes (activity feed entries, SignalR notifications, search index updates).
Templates (Phase 4)
Template Overview
Templates provide pre-built starting points for creating characters and factions. Instead of starting with a blank form, creators can select a template that pre-fills metadata fields, default properties, and structural elements appropriate for their game system or genre.
Template Structure
A template is a read-only definition that specifies default values for an entity type.
Template record:
| Field | Type | Notes |
|---|---|---|
| Id | Guid | Template identifier |
| Name | string | Template name (e.g., "D&D 5e Character") |
| Description | string | What this template is for |
| EntityType | enum | character, faction |
| Category | string | Grouping label (e.g., "D&D", "World of Darkness", "Generic") |
| DefaultMetadata | jsonb | Pre-filled metadata fields |
| DefaultProperties | jsonb | Pre-filled standard properties |
| IsSystemTemplate | bool | True for platform-provided templates, false for user-created |
| CreatedBy | Guid | Null for system templates, user ID for user-created |
| ProjectId | Guid | Null for system templates, project ID for user-created |
Example character templates:
| Template Name | Category | Default Metadata |
|---|---|---|
| D&D 5e Character | D&D | class, race, level, alignment, background, hit_points, armor_class |
| Vampire: The Masquerade | World of Darkness | clan, generation, sire, humanity, blood_potency |
| Generic Fantasy Character | Generic | race, occupation, alignment, notable_traits |
Example faction templates:
| Template Name | Category | Default Metadata |
|---|---|---|
| Noble House | Generic | motto, seat_of_power, founding_date, current_head |
| Criminal Organization | Generic | territory, front_business, hierarchy_type, specialization |
| Religious Order | Generic | deity, doctrine, holy_site, initiation_rites |
Template Usage Flow
Template application rules:
- The template provides default values; the user can override any field in the creation request
- Fields not overridden retain the template's default values
- The resulting entity is a regular character or faction with no ongoing link to the template -- changes to the template do not affect previously created entities
- The
templateIdis recorded on the created entity for reference but has no functional effect after creation
User-Created Templates (Future Consideration)
In a future iteration, users will be able to save an existing character or faction as a template. This involves:
- Extracting the metadata structure and current values from the entity
- Creating a new template record with
IsSystemTemplate = falseandCreatedByset to the user - Scoping the template to the user's project (other project members can use it, but it is not globally available)
This is a future enhancement and not part of the initial Phase 4 implementation.
Navigation Aids
Bookmarks
Users can bookmark any aggregate root entity (timeline update, character, faction) across any project they are a member of. Bookmarks are personal -- each user manages their own bookmark list.
Bookmark record:
| Field | Type | Notes |
|---|---|---|
| Id | Guid | Bookmark identifier |
| UserId | Guid | FK to User |
| ProjectId | Guid | FK to Project |
| EntityId | Guid | FK to the bookmarked entity |
| EntityType | enum | timeline_update, character, faction |
| CreatedAt | DateTime | When the bookmark was created |
Behavior:
- Adding a bookmark is idempotent -- bookmarking the same entity twice has no effect
- If a bookmarked entity is deleted, the bookmark is orphaned and excluded from query results (lazy cleanup on next bookmark list request)
- If the user loses access to a project (membership removed), their bookmarks for that project are excluded from responses but not deleted (in case membership is restored)
- Bookmarks are returned with the entity's current title and project key for display
Recently Viewed
The system tracks the last N entities a user has viewed, providing a "recently viewed" list for quick navigation.
RecentlyViewed record:
| Field | Type | Notes |
|---|---|---|
| Id | Guid | Record identifier |
| UserId | Guid | FK to User |
| ProjectId | Guid | FK to Project |
| EntityId | Guid | FK to the viewed entity |
| EntityType | enum | timeline_update, character, faction |
| ViewedAt | DateTime | When the entity was viewed |
Behavior:
- A view is recorded whenever a user fetches the detail endpoint for an aggregate root (e.g.,
GET /projects/{key}/characters/{seq}) - If the entity already exists in the user's recent list, the
ViewedAttimestamp is updated (upsert) - The recent list is capped at 50 entries per user; when the cap is reached, the oldest entry is evicted
- Visibility filtering applies -- if a user can no longer see an entity (e.g., a timeline update was unpublished), it is excluded from the recent list response
- Recently viewed items are returned ordered by
ViewedAtdescending (most recent first)
Pagination
The platform uses two pagination strategies depending on the use case:
Cursor-based pagination is used for feeds and lists where items may be added or removed between requests:
- Activity feed
- Search results
- Recently viewed items
Cursor-based pagination uses an opaque cursor (base64-encoded composite of the sort field value and entity ID) and a limit parameter. The response includes nextCursor (null if no more results) and hasMore boolean.
Offset-based pagination is used for stable, ordered collections:
- Timeline updates within a project
- Characters list
- Factions list
- Bookmarks
Offset-based pagination uses page and pageSize parameters. The response includes totalCount, page, pageSize, and totalPages.
Both strategies cap pageSize / limit at 100 items per request.
Data Flow
Search Query Pipeline -- Full Sequence
Export Generation Flow
Key Components
SearchService
The primary service for search operations. Accepts a query string and filter parameters, builds the appropriate tsquery, executes the full-text search against the SearchIndex table with permission-aware joins, and returns ranked, highlighted results. Encapsulates all PostgreSQL-specific search logic behind the ISearchEngine interface to support future migration to Meilisearch.
SearchIndexService
Responsible for maintaining the SearchIndex table. Listens to domain events (entity created, updated, deleted) and upserts or removes corresponding index rows. Handles tsvector construction with appropriate weight assignments per field. Provides a full reindex capability for rebuilding the index from scratch if needed.
ExportService
Manages the export job lifecycle. Creates job records, delegates file generation to format-specific generators (JsonExportGenerator, PdfExportGenerator), and handles status transitions. The service coordinates with IPermissionService to ensure the exported data is filtered by the requesting user's visibility.
ExportCleanupJob
A periodic background job that queries for export records past their ExpiresAt timestamp, deletes the associated files from storage, and marks the job records as expired. Runs on a configurable schedule (default: hourly).
TemplateService
Manages template retrieval and application. Lists available templates (system templates plus user-created templates for accessible projects), retrieves template details, and merges template defaults with user-provided overrides when creating an entity from a template.
BookmarkService
Manages per-user bookmarks. Handles creation (idempotent), deletion, and listing with entity resolution (fetching current titles and verifying the user still has access to the bookmarked project and entity).
RecentlyViewedService
Tracks entity views per user. Called by detail endpoints to record views, manages the per-user cap (evicting oldest entries), and provides the recently viewed list with visibility filtering.
BulkOperationService
Coordinates bulk status changes for timeline updates. Validates all transitions against the status state machine, checks permissions, and executes changes within a single database transaction. Raises individual domain events for each changed entity.
Feature Interactions
| Feature | Interaction |
|---|---|
| Authentication & User Management | All search, export, bookmark, and template endpoints require authentication. User identity drives bookmark ownership and recently viewed tracking. |
| Project Management | Search is scoped by project. Cross-project search requires project membership lookup. Export operates on a single project. Templates can be project-scoped. |
| Content Timeline System | Timeline updates, sections, and items are indexed for search. Bulk status changes operate on timeline updates. Timeline status determines search result visibility for Players/Viewers. |
| Character System | Characters are indexed for search. Character templates create new characters. Character visibility (public/private) affects search result filtering. |
| Faction System | Factions and faction relationships are indexed for search. Faction templates create new factions. |
| Permission & Visibility | Permission filtering is applied to every search query, export generation, bookmark resolution, and recently viewed list. The same IPermissionService rules that govern API responses govern search results. |
| Real-time & Collaboration | Export completion could trigger a SignalR notification to the requesting user. Bulk status changes trigger SignalR updates to connected project members. |
| Version History & Changelog | Bulk status changes generate version history entries for each affected timeline update. |
Edge Cases & Error Handling
Search Edge Cases
Empty query string: Return 400 Bad Request. A search query is required.
Query too long: Queries exceeding 200 characters are truncated to 200 characters before processing. No error is returned.
Special characters in query: PostgreSQL tsquery special characters (&, |, !, (, ), :, *) are stripped from the input to prevent injection and syntax errors.
No results: Return 200 OK with an empty results array and zero facet counts. This is not an error condition.
Search returns entity the user can no longer see: Permission filtering is applied at query time, so this should not happen. However, if a race condition causes a stale index entry to match (e.g., a character was made private between index update and query), the permission join in the query excludes it.
Deleted entity still in index: If an entity is deleted but the index row removal failed (e.g., event handler error), the search query joins against the source entity table. Missing source rows naturally exclude the result. A periodic reconciliation job can clean up orphaned index entries.
Export Edge Cases
Export requested for very large project: Export generation may take significant time. The async job pattern handles this gracefully. A maximum export size limit (e.g., 500MB) prevents runaway file generation. If exceeded, the job fails with an appropriate error message.
User loses project access during export: The export job records the user's role at the time of request. If membership is revoked while the job is processing, the generated file still reflects the permissions at request time. The download endpoint re-validates project membership, so if the user has been removed, the download returns 403 Forbidden even if the file exists.
Concurrent export requests: A user can have at most one active (requested or processing) export job per project. Submitting a new export request while one is active returns 409 Conflict with the existing job ID.
Export file storage failure: If writing the file to storage fails, the job transitions to failed status with an error message. The user can retry by creating a new export request.
Bulk Operation Edge Cases
Empty sequence list: Return 400 Bad Request. At least one timeline update must be specified.
Sequence number not found: If any sequence number does not exist in the project, return 422 with details about the missing sequences.
Mixed valid and invalid transitions: The entire batch is rejected. The response lists each sequence number with its current status and whether the requested transition is valid, so the user can correct and retry.
Concurrent bulk operations: Database-level row locking (SELECT FOR UPDATE) prevents two simultaneous bulk operations from conflicting. The second request waits for the first to complete.
Template Edge Cases
Template not found: Return 404 Not Found.
Template for wrong entity type: If a character template ID is used in the faction creation endpoint (or vice versa), return 400 Bad Request.
Override fields not in template: User-provided overrides for metadata fields not defined in the template are accepted and merged into the metadata. Templates define starting structure, not constraints.
Bookmark Edge Cases
Bookmark entity that was deleted: The bookmark remains in the database but is excluded from list responses. If the user explicitly requests a stale bookmark, return 404 for the underlying entity.
Bookmark across project the user left: Bookmarks for projects the user is no longer a member of are excluded from list responses but not deleted.
Recently Viewed Edge Cases
View tracking for non-aggregate-root entities: Only aggregate roots (timeline updates, characters, factions) are tracked. Viewing a section or item detail does not create a recently viewed entry (though viewing the parent timeline update does).
High-frequency views: If a user refreshes the same entity repeatedly, each view updates the ViewedAt timestamp but does not create duplicate entries (upsert on userId + entityId).
Phase & Priority
Phase 1: Core Search
- Full-text search within a project (
GET /api/projects/{key}/search?q=keyword) - Search index table and GIN index
- Index maintenance via domain events
- Permission-aware result filtering
- Keyword highlighting and relevance ranking
- Unified search result format
- Cursor-based pagination for search results
Phase 1: Navigation Aids
- Bookmarks (create, delete, list)
- Recently viewed tracking
- Pagination patterns (cursor-based and offset-based) used across all list endpoints
Phase 4: Advanced Search
- Filtered search (by type, status, creator, date range)
- Faceted result counts
- Cross-project search
Phase 4: Export
- Async export job pattern
- JSON export
- PDF export
- Export file cleanup
Phase 4: Bulk Operations
- Bulk status change for timeline updates
- Transactional all-or-nothing execution
Phase 4: Templates
- System-provided character and faction templates
- Create entity from template with override support
- Template listing and browsing
- User-created templates (future consideration beyond initial Phase 4)