Convex Data Migration

Import XILO data from backups into Convex with proper ID remapping. This guide covers the migration runner script and its usage.

Overview

The XILO migration runner (run-xilo-migration.ts) imports XILO-related data from a Convex backup ZIP file while maintaining referential integrity through ID remapping. It ensures that foreign key relationships between tables are preserved when data is imported into a new Convex deployment.

Prerequisites

Before running the migration:

1. TARGET backup already imported - Use npx convex import to import the TARGET backup first
2. Convex dev server running - Start with npx convex dev
3. Migration functions deployed - Deploy xiloMigration.ts with npx convex deploy

Usage

npx tsx scripts/convex-merge/run-xilo-migration.ts <xilo-backup.zip> [options]

Options

Option	Description
--dry-run	Preview the migration without importing data
--verbose	Show detailed error messages

Examples

# Preview migration (recommended first step)
npx tsx scripts/convex-merge/run-xilo-migration.ts ./convex_backups/xilo_PROD.zip --dry-run

# Run actual migration
npx tsx scripts/convex-merge/run-xilo-migration.ts ./convex_backups/xilo_PROD.zip

# Run with verbose error output
npx tsx scripts/convex-merge/run-xilo-migration.ts ./convex_backups/xilo_PROD.zip --verbose

Environment Variables

Configure these in your .env.local file:

Variable	Required	Description
CONVEX_URL	Yes*	Convex deployment URL
NEXT_PUBLIC_CONVEX_URL	Yes*	Alternative Convex URL
CONVEX_API_URL	Yes*	Alternative API URL (auto-converted)
NEXT_PUBLIC_APP_URL	No	New base URL for rewriting xiloUrl fields

*At least one Convex URL variable must be set.

Migration Phases

The migration runs in four sequential phases to handle table dependencies:

Phase 1: Base Tables (No Dependencies)

Tables imported first as they have no foreign key dependencies:

• missions - Mission records
• xilo - XILO video records

Phase 2: Single Dependency Tables

Tables that depend on Phase 1 tables:

Table	Depends On
scenes	missions
xiloAnnotations	xilo
xiloViews	xilo
xiloComments	xilo
xentralFavs	xilo
xentralHashtags	xilo

Phase 3: Multiple Dependency Tables

Tables with complex dependencies:

Table	Depends On
sceneVersions	scenes
xiloVersioning	xilo, scenes, missions, xiloAnnotations

Phase 4: Reference Updates

Updates existing TARGET table references with new XILO IDs:

• issueXiloCell - Remaps xiloVideoId and xiloAnnotationId strings
• issueXiloCellHistory - Remaps xiloVideoId and xiloAnnotationId strings
• Updates xiloUrl fields with new base URL (if NEXT_PUBLIC_APP_URL is set)

ID Remapping

The migration maintains mappings between old and new IDs for these tables:

missions       → old ID → new ID
xilo           → old ID → new ID
scenes         → old ID → new ID
xiloAnnotations → old ID → new ID
sceneVersions  → old ID → new ID

These mappings are passed to subsequent phases to remap foreign key references.

Tables Summary

Imported Tables

Table	Description
missions	Mission definitions
xilo	XILO video records
scenes	Scene definitions linked to missions
xiloAnnotations	Video annotations
xiloViews	View tracking
xiloComments	User comments
xentralFavs	User favorites
xentralHashtags	Hashtag associations
sceneVersions	Scene version history
xiloVersioning	XILO version history

Updated Tables (Existing)

Table	Fields Updated
issueXiloCell	xiloVideoId, xiloAnnotationId, xiloUrl
issueXiloCellHistory	xiloVideoId, xiloAnnotationId, xiloUrl

Skipped Tables

The following tables are not included in the migration:

• remixProjects
• remixSlides
• Other Remix-related tables

Add them to the migration script if needed.

Workflow

Recommended migration workflow:

1. Export TARGET backup     → npx convex export --path ./target-backup.zip
2. Export XILO backup       → npx convex export --path ./xilo-backup.zip
3. Import TARGET backup     → npx convex import ./target-backup.zip
4. Deploy migration funcs   → npx convex deploy
5. Dry run XILO migration   → npx tsx run-xilo-migration.ts ./xilo-backup.zip --dry-run
6. Run XILO migration       → npx tsx run-xilo-migration.ts ./xilo-backup.zip
7. Verify data              → Check Convex dashboard

Troubleshooting

Common Errors

"No Convex URL found in environment"

• Ensure one of CONVEX_URL, NEXT_PUBLIC_CONVEX_URL, or CONVEX_API_URL is set in .env.local

"Backup file not found"

• Verify the path to your backup ZIP file is correct
• Use absolute paths if relative paths cause issues

Records failing to import

• Run with --verbose flag to see detailed error messages
• Check that prerequisite tables were imported first
• Verify the backup file isn't corrupted

Verification

After migration, verify data in the Convex dashboard:

1. Check record counts match expected values
2. Verify foreign key references resolve correctly
3. Test xiloUrl links work with the new base URL
4. Check issueXiloCell records link to valid XILO videos