Skip to main content

Database Change Log

A record of the schema and data changes the Go backend applies to a RAGFlow database, and of the version or schema checks that decide which of them still have to run.

How the database version is recorded

  • The marker is a row in system_settings with the key mysql_migration.database.version.
  • For version-gated steps, the marker prevents a migration step from running again after that step has completed successfully.
  • Go defines the versions as constants: modelMigrationBaseVersion and modelMigrationTargetVersion in migration_version.go, conversationHistoryTargetVersion in conversation_history_migration.go, knowledgebaseParserConfigTargetVersion in migration.go, and ingestionParserConfigTargetVersion in ingestion_parser_config_migration.go. The two parser-config constants share v1.0.0-rc2. The DSL multi-version backfill is gated by dslVersionTargetVersion (v1.0.0-rc3.dev6) in dsl_version_migration.go.
  • The current permission migration targets permissionMigrationTargetVersion (v1.0.0-rc3.dev4) in permission_migration.go. Tenant default-model migration targets tenantDefaultModelsTargetVersion (v1.0.0-rc3.dev5) in tenant_default_models_migration.go.
  • The compilation queue split records knowledgeCompileQueueTargetVersion (v1.0.0-rc3.dev5), defined in knowledge_compile_queue_migration.go, after both queues finish. Source-column presence controls the queue copy even when the marker already contains this or a newer version.
  • A version-gated step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a dev prerelease sorts before the release it precedes (v1.0.0-rc1.dev1 < v1.0.0-rc1). A missing or unparsable marker never skips a step.
  • checkDatabaseVersion in cmd/ragflow_server.go refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.

Startup order in InitDB for ragflow_server --migrate (database.go):

  1. RunMigrations — manual, idempotent steps plus the two versioned tenant model steps. Runs before AutoMigrate because it has to observe the legacy schema.
  2. migrateIngestionLogRunIdentity — ingestion log columns and indexes.
  3. migrateUserTenantDuplicates — deduplicates memberships before AutoMigrate creates their unique index.
  4. AutoMigrate — converges every ORM model, including the compilation queue tables.
  5. migrateConversationHistory — runs after AutoMigrate, which creates the child tables it backfills.
  6. migrateIngestionParserConfigs — converts ingestion settings on datasets and documents, then invokes the existing migrateKnowledgebaseParserConfig cleanup. Both run after the conversation split so the shared marker cannot skip that split.
  7. migrateTenantModelMaxTokens — backfills model context windows before advancing the marker to its target.
  8. migratePermissions — applies the cumulative permission migration through v1.0.0-rc3.dev4. The enterprise hook runs before the shared version check.
  9. migrateTenantDefaultModels — moves tenant-wide model selections to owner memberships, drops the tenant columns, and records v1.0.0-rc3.dev5.
  10. migrateTenantModelProviderUserUniqueIndex — reconciles provider ownership indexes.
  11. migrateKnowledgeCompileQueues — copies the existing compilation queues and drops their source columns, then records v1.0.0-rc3.dev5 if the marker is older. It runs after tenant default-model migration so the queue split cannot make that step skip its data changes. Queue copying remains gated on source-column presence even when the preceding step has already recorded dev5.
  12. MigrateDSLVersionColumns and BackfillDSLVersionBindings — add pipeline version columns and bind existing dataflow canvases, then record v1.0.0-rc3.dev6.
  13. migrateConversationReferences — move answer references into message rows and record v1.0.0-rc3.dev7.

Version overview

Database versionScopeGate
v0.26.0Rebuild the tenant model tables from tenant_llm and normalize stored model idsmodelMigrationBaseVersion
v1.0.0-rc1Seed factory-declared models, merge model_type into an integer bitmask, populate the tenant_*_id columnsmodelMigrationTargetVersion
v1.0.0-rc1Split conversation message and reference payloads into child tablesconversationHistoryTargetVersion
v1.0.0-rc2Convert Python ingestion settings in dataset/document parser_config onto pipeline components, then remove retired dataset raptor / graphrag sectionsingestionParserConfigTargetVersion / knowledgebaseParserConfigTargetVersion
v1.0.0-rc3.dev1Permission ownership columns, nullable inviters, and unique tenant membershipsHistorical permission target; included in the current cumulative permission migration
v1.0.0-rc3.dev2No standalone community migration is defined in the checked-in code or its historyNo separate community gate
v1.0.0-rc3.dev3Tenant role support, user-scoped model providers, and enterprise migration-marker separationHistorical permission target; provider index reconciliation uses schema checks
v1.0.0-rc3.dev4Normalize legacy tenant invitation roles and statusespermissionMigrationTargetVersion
v1.0.0-rc3.dev5Move tenant default-model selections to owner memberships and drop the tenant columnstenantDefaultModelsTargetVersion
v1.0.0-rc3.dev5Split compilation queues into knowledge_compile_docs_backlog and knowledge_compile_docs_inflight, then drop their source columnsknowledgeCompileQueueTargetVersion; source-column presence controls copying
v1.0.0-rc3.dev6Lock existing dataflow-canvas dataset/document bindings to the canvas latest version (the pipeline_version_id columns are added by the version-less step below)dslVersionTargetVersion
v1.0.0-rc3.dev7Store conversation references in each assistant messageconversationReferenceTargetVersion

The two tenant model steps are cumulative: a database at v0.26.0 still runs the v1.0.0-rc1 step, and a database at or above v1.0.0-rc1 runs neither.

The dev1–dev4 permission entries below describe the development history. They are consolidated into the current migratePermissions body targeting dev4; they are not four independent migration functions. The two dev5 changes run in the order listed above and share the database version marker. A dev5 marker alone does not prove that the compilation queue columns have been removed.


v0.26.0

Schema changes

ObjectChange
system_settingsCreated if missing, so the version marker can be written before AutoMigrate reaches the table.
tenant_model_providerCreated if missing. Columns: id (PK, varchar 32), provider_name, tenant_id, base timestamps. Unique index idx_tenant_provider_unique (tenant_id, provider_name).
tenant_model_instanceCreated if missing. Columns: id (PK), instance_name, provider_id, api_key, status, extra, base timestamps.
tenant_modelCreated if missing. Columns: id (PK), model_name, provider_id, instance_id, model_type, status, extra, base timestamps.

The three tables are created only when the legacy tenant_llm table exists and the target table does not, mirroring the CREATE TABLE IF NOT EXISTS the Python stages run. Existing tables are never altered by this step: tenant_model.model_type deliberately keeps the text shape the Python migration left behind so the v1.0.0-rc1 merge can still read the legacy model names.

Data migration

TableChange
tenant_model_providerOne row per distinct (tenant_id, llm_factory) in tenant_llm; provider_name takes the factory name.
tenant_model_instanceOne row per (tenant_id, llm_factory), carrying the first api_key after deduplication; instance_name defaults to default.
tenant_modelRows derived from tenant_llm: model_type is written as an integer bitmask, only groups whose merged status is active are kept, and provider_id / instance_id point at the rows created above. Bits: chat=1, embedding=2, asr/speech2text=4, vision/image2text=8, rerank=16, tts=32, ocr=64.
tenant, knowledgebase, dialog, memoryStored model ids are rewritten from <model>@<provider> to <model>@default@<provider>: tenant.llm_id / embd_id / asr_id / img2txt_id / rerank_id / tts_id / ocr_id, knowledgebase.embd_id, dialog.llm_id / rerank_id, memory.embd_id / llm_id, plus the same keys nested inside parser_config and prompt_config JSON (llm_id, embd_id, embedding_model, rerank_id, asr_id, img2txt_id, tts_id, ocr_id).

On success the step writes v0.26.0 into the version marker.


v0.27.2

Schema changes

TableChange
tenant_modelmodel_type is converted from the legacy text representation to the integer bitmask (see the data migration below). Once the column is an integer the conversion is a no-op.
tenantAdds tenant_llm_id, tenant_embd_id, tenant_asr_id, tenant_img2txt_id, tenant_rerank_id, tenant_tts_id, tenant_ocr_id — each varchar(32) NULL.
knowledgebaseAdds tenant_embd_id — varchar(32) NULL.
dialogAdds tenant_llm_id, tenant_rerank_id — varchar(32) NULL.
memoryAdds tenant_embd_id, tenant_llm_id — varchar(32) NULL.

The tenant_*_id columns are added only when missing, and are skipped entirely when any of tenant_model, tenant_model_provider, tenant_model_instance is absent.

Data migration

TableChange
tenant_modelSeeding: the models declared in conf/llm_factories.json are inserted for every provider/instance created by the v0.26.0 step. When a (provider_id, instance_id, model_name) row already exists, the declared factory bits are OR-ed into its model_type instead of a second row being inserted.
tenant_modelType merge: rows sharing (provider_id, instance_id, model_name) collapse into a single row whose model_type is the OR of the contributing bits. Bits that only ever appeared on unsupported rows are dropped, and groups whose merged status is not active are dropped.
tenant, knowledgebase, dialog, memoryId backfill: each tenant_*_id column is filled from its legacy counterpart (llm_id, embd_id, …) by resolving the referenced tenant_model.id through a (tenant_id, model_name, provider, model_type) lookup over active models, with the model_type bitmask expanded back into individual types.

On success the step writes v0.27.2 into the version marker.


v1.0.0-rc1

This step only prepares the schema owned by v1.0.0-rc1; dropping the legacy payload columns is out of its scope.

Schema changes

TableChange
conversation_messageCreated by AutoMigrate. Composite PK (conversation_id, position); columns message_id, role, content, content_type, status, thumb_up, feedback, created_at, metadata; composite index message_lookup (conversation_id, message_id, role, position); foreign key conversation_id → conversation.id with ON DELETE CASCADE.
conversation_referenceCreated by AutoMigrate. Composite PK (conversation_id, position); columns reference (longtext), message_position; foreign key conversation_id → conversation.id with ON DELETE CASCADE.
api_4_conversation_messageSame shape as conversation_message, parented by api_4_conversation.
api_4_conversation_referenceSame shape as conversation_reference, parented by api_4_conversation.
conversation, api_4_conversationUnchanged by this step: the legacy message and reference payload columns stay in place. The Go schema marks both gorm:"-", so only a probe model can still read them.

Data migration

TableChange
conversation_message, api_4_conversation_messageEach element of the parent row's message JSON array becomes one row, position being its index in the array.
conversation_reference, api_4_conversation_referenceEach entry of the parent row's reference JSON array becomes one row, position being its index in the array. message_position stays NULL: the payload carries no message binding, and only references written together with a message by the current code path set it.

Execution details:

  • Conversations are processed in cursor-ordered batches of 100, each batch in its own transaction, so an interrupted run resumes from the last committed batch.
  • A conversation that already has rows in either child table is left untouched: rows written after the split took effect win over the payload, so a rerun neither duplicates nor clobbers them.
  • Parent tables without message and reference columns are skipped.

On success the step writes v1.0.0-rc1 into the version marker.


v1.0.0-rc2

Converts saved Python v0.27.2 ingestion settings into the component-scoped configuration consumed by Go. After conversion, the existing dataset cleanup removes the retired raptor and graphrag sections.

Data migration

TableChange
knowledgebase, documentMove supported top-level Parser, Chunker, Tokenizer and Extractor settings onto the corresponding pipeline component nodes.
knowledgebaseThe existing cleanup deletes top-level raptor and graphrag sections, preserving other settings.

The conversion covers parser engines and model references, PDF pages, MinerU method/language, TCADP options, spreadsheet column configuration, supported chunk sizes and delimiters, overlap, media context, parent/child settings, filename embedding weight, keywords, questions and metadata. Fields are converted only where the selected component supports them. Settings without a Go consumer are dropped with table/row/field warnings; tags and early Go node formats are left untouched.

Execution details:

  • Component conversion reads each table in id-ordered batches of 256, each batch in its own transaction. Changed rows are updated once by the conversion; unchanged rows retain their original bytes. The existing dataset cleanup performs its own subsequent scan.
  • Each row's parser_id determines the builtin component targets. Parser/Chunker/Tokenizer conversion leaves custom pipeline rows and unsupported parser IDs untouched. Extractor conversion can also target existing Extractor nodes in custom pipeline configs.
  • Existing component values win, except that dataset-level modular metadata transport remains authoritative under the current service contract.
  • SQL NULL, empty strings and JSON null are left untouched. Invalid JSON, unsupported metadata schemas and unsupported engine selections stop migration with table/row context. A failed batch rolls back; earlier committed batches can be rerun safely. See Parser-config conversion failures for recovery instructions.
  • The marker advances only after both table conversions and the existing dataset cleanup succeed. If neither table exists, the marker is left unchanged.

On success the migration records v1.0.0-rc2. Databases already marked v1.0.0-rc2 or later skip conversion. This includes databases marked by the earlier cleanup-only implementation: that marker does not prove their ingestion settings were converted.


v1.0.0-rc3.dev1

Permission schema and ownership

The initial permission migration introduced these changes, now retained in the cumulative migratePermissions implementation:

TableSchema and data change
user_tenantMakes invited_by nullable, widens status to varchar(16), and adds unique index idx_user_tenant_user_tenant (user_id, tenant_id).
tenant_model_providerAdds user_id varchar(32) and fills missing or empty owners from tenant_id.
api_tokenAdds nullable user_id varchar(32) and fills missing or empty owners from tenant_id.
user_canvasAdds nullable tenant_id varchar(32) and fills missing or empty tenant IDs from user_id.
tenantAdds invite_enabled with default false and nullable role_id.

Before AutoMigrate creates the membership unique index, migrateUserTenantDuplicates merges duplicate (user_id, tenant_id) memberships. It retains the preferred joined membership using role and timestamp ordering, remaps references in existing group_info, group_member, department_member, and permission tables, and deletes the duplicates. This schema-checked preparation runs independently of the version marker.

The ownership backfills preserve non-empty values. They run after schema creation through migratePermissions, which currently records dev4 after its cumulative work succeeds.


v1.0.0-rc3.dev2

No standalone community schema or data migration for dev2 is defined in this checkout. The community permission target moved directly from dev1 to dev3; dev2 is exercised as an existing database version in the permission migration tests. Upgrading from a dev2 marker runs the cumulative permission migration targeting dev4.

Enterprise permission work is supplied through permissionDataModelsEE and migratePermissionDataEE. This community checkout provides empty/no-op implementations, so it does not define enterprise-only dev2 changes.


v1.0.0-rc3.dev3

Tenant roles and provider ownership

Table or migrationChange
userAdds nullable, indexed role_id for user role assignment.
tenant_model_providerReplaces uniqueness on (provider_name, tenant_id) with idx_tenant_provider_user_unique (provider_name, tenant_id, user_id), allowing different users in one tenant to own providers with the same name.
migratePermissionsInvokes the enterprise permission hook before checking the shared version marker, so a completed community migration cannot suppress enterprise data work. Enterprise implementations can use their own marker through the marker-specific read/write helpers.

Provider index reconciliation is implemented in tenant_model_provider_migration.go. It checks existing indexes by column set, creates the replacement before dropping old two-column unique indexes, and runs independently of the permission version gate. Enterprise system-setting initialization also runs after the migration/server schema setup; it is a no-op in this community checkout.


v1.0.0-rc3.dev4

Legacy tenant invitations

migrateLegacyTenantInvitationStatuses, called by migratePermissions, rewrites user_tenant rows whose role is invite and whose status is joined ('1') or SQL NULL. The resulting role is an empty string and status is invited ('2'). Other memberships remain unchanged.

This step adds no separate schema object. permissionMigrationTargetVersion is now v1.0.0-rc3.dev4; success records that value after the invitation repair and earlier permission backfills finish. A database already at dev4 or later skips the shared permission body, while the enterprise hook still runs before that check.


v1.0.0-rc3.dev5

Tenant default models

Implemented by tenant_default_models_migration.go, with target tenantDefaultModelsTargetVersion (v1.0.0-rc3.dev5).

TableSchema and data change
user_tenantReceives the model-selection columns through AutoMigrate. Missing or empty selections are filled from the matching tenant row only for memberships with role owner or user_id = tenant_id. Existing selections and other members' settings are preserved.
tenantDestructive: drops the old model-selection columns after the transactional copy succeeds.

The columns are llm_id, embd_id, asr_id, img2txt_id, rerank_id, tts_id, ocr_id, and decision_id, together with their corresponding tenant_*_id columns. Copying all columns is transactional; column drops follow and check whether each column still exists, allowing an interrupted run to resume before the marker is advanced.

This version-gated step skips databases already marked dev5 or later. When either required table is absent, there is no copy or drop, but the step still records its target version. It runs before the queue split, which also targets dev5.

Knowledge compilation queues

Introduced in v1.0.0-rc3.dev5 and implemented by migrateKnowledgeCompileQueues in knowledge_compile_queue_migration.go. Each queue is migrated when its source column exists, regardless of mysql_migration.database.version. After both queues finish, the step records knowledgeCompileQueueTargetVersion (v1.0.0-rc3.dev5) in that marker if its current value is older. A newer version is preserved.

TableSchema and data change
knowledge_compile_docs_backlogCreated by AutoMigrate. Each event in knowledge_compile_docs.backlog_doc_ids becomes a row with dataset_id, doc_id, queue_position, event_type, variants, and task_types.
knowledge_compile_docs_inflightCreated by AutoMigrate with the same columns; populated from knowledge_compile_docs.inflight_doc_ids.
knowledge_compile_docsDestructive: drops backlog_doc_ids and inflight_doc_ids, each after its queue has been copied successfully. Dataset scheduling state, leases, progress, and diagnostics remain in this table.

Both new tables have no id, primary key, or unique constraint. A non-unique index on (dataset_id, queue_position) supports ordered reads. Migration preserves duplicate events, array order, and routing metadata; variants and task_types remain JSON arrays stored in longtext columns.

Datasets are read in batches of 100, with each dataset copied in its own transaction. Info-level progress logs identify the queue and batch range, report committed dataset and entry counts and remaining datasets, and record each source-column drop. Entry counts include duplicate events and reset for each queue and migration run. Failed dataset copies roll back and log the dataset and committed progress. Rerunning while services remain stopped replaces copied rows for any source column that still exists, so retries do not duplicate events.

The queue step advances the version marker only after both queue copies and source-column drops succeed. If either queue fails, it leaves the marker at its value on entry; the preceding tenant default-model step may already have recorded dev5. The source-column checks and server startup guard therefore remain authoritative for queue migration completion. Databases whose queues were already split also advance to v1.0.0-rc3.dev5 without copying entries again when their marker is older. The completion log includes the database version and target version.

This migration requires a coordinated service stop:

  1. Stop all services that publish or consume compilation events.
  2. Run ragflow_server --migrate --log-level info to see progress and wait for successful completion of both queues.
  3. Restart the services.

Starting a server while either source column remains fails with knowledge compilation queues need migration. A database version marker at or above the current release does not bypass this requirement. If migration fails, keep services stopped, resolve the reported error, and rerun ragflow_server --migrate.

v1.0.0-rc3.dev6

Locks every pre-existing dataflow-canvas binding to the latest version of that canvas, so already-configured datasets and documents keep parsing against the version that was current when the upgrade ran (design section 5/B4 sticky snapshot). The pipeline_version_id columns it writes are added by the version-less step below; this step only backfills their values and is gated by the dslVersionTargetVersion marker.

Data migration

TableChange
knowledgebase, documentRows whose pipeline_id points at a dataflow_canvas are bound to that canvas latest version (user_canvas_version ordered by create_time DESC). Bindings to a built-in parser id (no backing user_canvas), to a non-dataflow canvas, or to a canvas with no versions are left unbound; a dangling version id is never written.

Execution details:

  • Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes cleanly.
  • The marker advances to v1.0.0-rc3.dev6 only after both tables are processed. Databases already marked v1.0.0-rc3.dev6 or later skip the backfill.

On success the step writes v1.0.0-rc3.dev6 into the version marker.


v1.0.0-rc3.dev7

Adds a nullable reference LONGTEXT column to conversation_message and api_4_conversation_message. New chat, agent, and channel answers store their reference object directly on their message row.

Run ragflow_server --migrate with writers stopped before starting the new services. The migration copies conversation_reference and api_4_conversation_reference into the corresponding assistant messages. It uses message_position when present; older array entries follow assistant order after the first user message, excluding the prologue. Rows are read in batches of 100 and written in transactions. Reruns preserve reference objects already stored on messages. References duplicated in message metadata are moved into the column and removed from metadata, so generation reads avoid that data.

After both message tables are copied successfully, the migration drops conversation_reference and api_4_conversation_reference. Automatic schema creation no longer recreates these tables. Source-table presence triggers copying and cleanup even if the marker already contains v1.0.0-rc3.dev7 or a newer version; a newer marker is preserved.

References without a surviving assistant message are skipped and counted in a warning. These include references left behind by deleted messages, empty histories, and unsuccessful turns. They are removed with the source tables and are never attached to another answer. Invalid JSON or a copy error stops the migration before either source table is dropped. Resolve the reported issue and rerun it. Completed batches are safe to retry. The marker advances to v1.0.0-rc3.dev7 only after both copies and table drops finish. A failed table drop can be retried even if the other source table has already been dropped.

Session responses return reference within each answer, with no session-level reference array when include_reference=true. Clients must read messages[i].reference (or message[i].reference for agent sessions). include_reference and include_prompt both default to false; omitted fields are excluded from SQL as well as the response. The UI requests the fields it displays. Generation history SQL omits reference, prompt, usage, and elapsed_ms; session detail reads request the fields they display.


Version-less steps

These steps are not gated by the version marker. They use schema or data checks to make repeated execution safe. Steps in RunMigrations run during ragflow_server --migrate; ingestion helpers also run during server startup.

Schema changes

TableChange
tenant_llmComposite primary key (tenant_id, llm_factory, llm_name) is replaced by a surrogate auto-increment id; the legacy uniqueness is preserved by a unique index on the old key columns.
task, documentprocess_duation renamed to process_duration.
userUnique index on email.
ingestion_taskUnique index on document_id.
knowledgebaseAdds the VIRTUAL generated column name_ci (LOWER(name) for status='1' rows, NULL otherwise) and the unique index idx_kb_tenant_name_ci (tenant_id, name_ci), so soft-deleted rows never block name reuse.
user_canvasUnique index (user_id, canvas_category, title).
pipeline_operation_logAdds run_count int NULL and the index idx_pipeline_operation_log_document_run.
ingestion_taskAdds pipeline_log_id varchar(32) NULL.
ingestion_task_logAdds pipeline_log_id varchar(32) NULL, event_type tinyint NOT NULL DEFAULT 4, and the index idx_ingestion_task_log_pipeline_id.
dialog, tenant_llm, api_token, canvas_template, system_settings, knowledgebaseColumn types converged to the Peewee models, for example dialog.top_k int NOT NULL DEFAULT 1024, tenant_llm.api_key text, api_token.dialog_id varchar(32), canvas_template.title / description nullable longtext, system_settings.value text NOT NULL, knowledgebase.raptor_task_finish_at / mindmap_task_finish_at datetime.
conversation, api_4_conversationIndex idx_<table>_dialog_updated (dialog_id, update_time, id) for conversation list queries.
knowledgebaseAdds pipeline_version_id varchar(32) NULL (dataflow canvas version binding; design section 5/B4).
documentAdds pipeline_version_id varchar(32) NULL (inherits the dataset pinned version at upload; design section 5).
user_canvas_versionAdds alias varchar(255) NULL (free-text version alias for the version list filter; design A2/F1).

Data changes

TableChange
knowledgebase, dialogparser_config entries stored under the legacy key TokenChunker:SixApplesFall are rewritten to GeneralChunker:SixApplesFall. Values already stored under the new key win; TokenChunker-only fields (delimiter_mode, outputs) are dropped, and a legacy delimiter becomes delimiters. delimiter is discarded when delimiters already exists.