Skip to main content

Database Change Log

A record of the schema and data changes the Go backend applies to a RAGFlow database, and of the version or schema checks that decide which of them still have to run.

How the database version is recorded

  • The marker is a row in system_settings with the key mysql_migration.database.version.
  • For version-gated steps, the marker prevents a migration step from running again after that step has completed successfully.
  • Go defines the versions as constants: modelMigrationBaseVersion and modelMigrationTargetVersion in migration_version.go, conversationHistoryTargetVersion in conversation_history_migration.go, knowledgebaseParserConfigTargetVersion in migration.go, and ingestionParserConfigTargetVersion in ingestion_parser_config_migration.go. The two parser-config constants share v1.0.0-rc2. The DSL multi-version backfill is gated by dslVersionTargetVersion (v1.0.0-rc3.dev6) in dsl_version_migration.go. Resource-owner IDs are backfilled through resourceUserIDMigrationTargetVersion (v1.0.0.1) in resource_user_id_migration.go. Legacy ingestion component params are normalized through ingestionLegacyConfigTargetVersion (v1.0.0.2) in ingestion_legacy_config_migration.go.
  • The current permission migration targets permissionMigrationTargetVersion (v1.0.0-rc3.dev4) in permission_migration.go. Tenant default-model migration targets tenantDefaultModelsTargetVersion (v1.0.0-rc3.dev5) in tenant_default_models_migration.go.
  • The compilation queue split records knowledgeCompileQueueTargetVersion (v1.0.0-rc3.dev5), defined in knowledge_compile_queue_migration.go, after both queues finish. Source-column presence controls the queue copy even when the marker already contains this or a newer version.
  • A version-gated step runs when the stored version is lower than the step's version. Versions are compared with Python PEP 440 semantics, where a dev prerelease sorts before the release it precedes (v1.0.0-rc1.dev1 < v1.0.0-rc1). A missing or unparsable marker never skips a step.
  • checkDatabaseVersion in cmd/ragflow_server.go refuses to start when the code version is older than the stored database version, so a database migrated by a newer release cannot be opened by an older one.

Startup order in InitDB for ragflow_server --migrate (database.go):

  1. RunMigrations — manual, idempotent steps plus the two versioned tenant model steps. This includes migrateColumnWidths, which widens pipeline_operation_log.pipeline_title, tenant.name, and the tenant_*id model-reference columns only when the existing column is narrower. Runs before AutoMigrate because it has to observe the legacy schema.
  2. migrateIngestionLogRunIdentity — ingestion log columns and indexes.
  3. migrateUserTenantDuplicates — deduplicates memberships before AutoMigrate creates their unique index.
  4. AutoMigrate — converges every ORM model, including the compilation queue tables.
  5. migrateConversationHistory — runs after AutoMigrate, which creates the child tables it backfills.
  6. migrateIngestionParserConfigs — converts ingestion settings on datasets and documents, then invokes the existing migrateKnowledgebaseParserConfig cleanup. Both run after the conversation split so the shared marker cannot skip that split.
  7. migrateTenantModelMaxTokens — backfills model context windows before advancing the marker to its target.
  8. migratePermissions — applies the cumulative permission migration through v1.0.0-rc3.dev4. The enterprise hook runs before the shared version check.
  9. migrateTenantDefaultModels — moves tenant-wide model selections to owner memberships, drops the tenant columns, and records v1.0.0-rc3.dev5.
  10. migrateTenantModelProviderUserUniqueIndex — reconciles provider ownership indexes.
  11. migrateKnowledgeCompileQueues — copies the existing compilation queues and drops their source columns, then records v1.0.0-rc3.dev5 if the marker is older. It runs after tenant default-model migration so the queue split cannot make that step skip its data changes. Queue copying remains gated on source-column presence even when the preceding step has already recorded dev5.
  12. MigrateDSLVersionColumns and BackfillDSLVersionBindings — add pipeline version columns and bind existing dataflow canvases, then record v1.0.0-rc3.dev6.
  13. migrateConversationReferences — move answer references into message rows and record v1.0.0-rc3.dev7.
  14. migrateResourceUserIDs — backfills resource user_id values from their owner columns and records v1.0.0.1. AutoMigrate creates the columns before this step runs.
  15. migrateIngestionLegacyConfigs — normalizes top-level flat and component-node legacy ingestion params in datasets, documents, and custom canvas DSLs, then records v1.0.0.2. It runs last because its target version sorts above every earlier gated step.

Version overview

Database versionScopeGate
v0.26.0Rebuild the tenant model tables from tenant_llm and normalize stored model idsmodelMigrationBaseVersion
v1.0.0-rc1Seed factory-declared models, merge model_type into an integer bitmask, populate the tenant_*_id columnsmodelMigrationTargetVersion
v1.0.0-rc1Split conversation message and reference payloads into child tablesconversationHistoryTargetVersion
v1.0.0-rc2Convert Python ingestion settings in dataset/document parser_config onto pipeline components, then remove retired dataset raptor / graphrag sectionsingestionParserConfigTargetVersion / knowledgebaseParserConfigTargetVersion
v1.0.0-rc3.dev1Permission ownership columns, nullable inviters, and unique tenant membershipsHistorical permission target; included in the current cumulative permission migration
v1.0.0-rc3.dev2No standalone community migration is defined in the checked-in code or its historyNo separate community gate
v1.0.0-rc3.dev3Tenant role support, user-scoped model providers, and enterprise migration-marker separationHistorical permission target; provider index reconciliation uses schema checks
v1.0.0-rc3.dev4Normalize legacy tenant invitation roles and statusespermissionMigrationTargetVersion
v1.0.0-rc3.dev5Move tenant default-model selections to owner memberships and drop the tenant columnstenantDefaultModelsTargetVersion
v1.0.0-rc3.dev5Split compilation queues into knowledge_compile_docs_backlog and knowledge_compile_docs_inflight, then drop their source columnsknowledgeCompileQueueTargetVersion; source-column presence controls copying
v1.0.0-rc3.dev6Lock existing dataflow-canvas dataset/document bindings to the canvas latest version (the pipeline_version_id columns are added by the version-less step below)dslVersionTargetVersion
v1.0.0-rc3.dev7Store conversation references in each assistant messageconversationReferenceTargetVersion
v1.0.0.1Backfill resource-owner user_id values from created_by or tenant_idresourceUserIDMigrationTargetVersion
v1.0.0.2Normalize legacy Extractor / TitleChunker / TokenChunker params in knowledgebase, document, user_canvas, and user_canvas_versioningestionLegacyConfigTargetVersion

The two tenant model steps are cumulative: a database at v0.26.0 still runs the v1.0.0-rc1 step, and a database at or above v1.0.0-rc1 runs neither.

The dev1–dev4 permission entries below describe the development history. They are consolidated into the current migratePermissions body targeting dev4; they are not four independent migration functions. The two dev5 changes run in the order listed above and share the database version marker. A dev5 marker alone does not prove that the compilation queue columns have been removed.


v0.26.0

Schema changes

ObjectChange
system_settingsCreated if missing, so the version marker can be written before AutoMigrate reaches the table.
tenant_model_providerCreated if missing. Columns: id (PK, varchar 32), provider_name, tenant_id, base timestamps. Unique index idx_tenant_provider_unique (tenant_id, provider_name).
tenant_model_instanceCreated if missing. Columns: id (PK), instance_name, provider_id, api_key, status, extra, base timestamps.
tenant_modelCreated if missing. Columns: id (PK), model_name, provider_id, instance_id, model_type, status, extra, base timestamps.

The three tables are created only when the legacy tenant_llm table exists and the target table does not, mirroring the CREATE TABLE IF NOT EXISTS the Python stages run. Existing tables are never altered by this step: tenant_model.model_type deliberately keeps the text shape the Python migration left behind so the v1.0.0-rc1 merge can still read the legacy model names.

Data migration

TableChange
tenant_model_providerOne row per distinct (tenant_id, llm_factory) in tenant_llm; provider_name takes the factory name.
tenant_model_instanceOne row per (tenant_id, llm_factory), carrying the first api_key after deduplication; instance_name defaults to default.
tenant_modelRows derived from tenant_llm: model_type is written as an integer bitmask, only groups whose merged status is active are kept, and provider_id / instance_id point at the rows created above. Bits: chat=1, embedding=2, asr/speech2text=4, vision/image2text=8, rerank=16, tts=32, ocr=64.
tenant, knowledgebase, dialog, memoryStored model ids are rewritten from <model>@<provider> to <model>@default@<provider>: tenant.llm_id / embd_id / asr_id / img2txt_id / rerank_id / tts_id / ocr_id, knowledgebase.embd_id, dialog.llm_id / rerank_id, memory.embd_id / llm_id, plus the same keys nested inside parser_config and prompt_config JSON (llm_id, embd_id, embedding_model, rerank_id, asr_id, img2txt_id, tts_id, ocr_id).

On success the step writes v0.26.0 into the version marker.


v0.27.2

Schema changes

TableChange
tenant_modelmodel_type is converted from the legacy text representation to the integer bitmask (see the data migration below). Once the column is an integer the conversion is a no-op.
tenantAdds tenant_llm_id, tenant_embd_id, tenant_asr_id, tenant_img2txt_id, tenant_rerank_id, tenant_tts_id, tenant_ocr_id — each varchar(32) NULL.
knowledgebaseAdds tenant_embd_id — varchar(32) NULL.
dialogAdds tenant_llm_id, tenant_rerank_id — varchar(32) NULL.
memoryAdds tenant_embd_id, tenant_llm_id — varchar(32) NULL.

The tenant_*_id columns are added only when missing, and are skipped entirely when any of tenant_model, tenant_model_provider, tenant_model_instance is absent.

Data migration

TableChange
tenant_modelSeeding: the models declared in conf/llm_factories.json are inserted for every provider/instance created by the v0.26.0 step. When a (provider_id, instance_id, model_name) row already exists, the declared factory bits are OR-ed into its model_type instead of a second row being inserted.
tenant_modelType merge: rows sharing (provider_id, instance_id, model_name) collapse into a single row whose model_type is the OR of the contributing bits. Bits that only ever appeared on unsupported rows are dropped, and groups whose merged status is not active are dropped.
tenant, knowledgebase, dialog, memoryId backfill: each tenant_*_id column is filled from its legacy counterpart (llm_id, embd_id, …) by resolving the referenced tenant_model.id through a (tenant_id, model_name, provider, model_type) lookup over active models, with the model_type bitmask expanded back into individual types.

On success the step writes v0.27.2 into the version marker.


v1.0.0-rc1

This step only prepares the schema owned by v1.0.0-rc1; dropping the legacy payload columns is out of its scope.

Schema changes

TableChange
conversation_messageCreated by AutoMigrate. Composite PK (conversation_id, position); columns message_id, role, content, content_type, status, thumb_up, feedback, created_at, metadata; composite index message_lookup (conversation_id, message_id, role, position); foreign key conversation_id → conversation.id with ON DELETE CASCADE.
conversation_referenceCreated by AutoMigrate. Composite PK (conversation_id, position); columns reference (longtext), message_position; foreign key conversation_id → conversation.id with ON DELETE CASCADE.
api_4_conversation_messageSame shape as conversation_message, parented by api_4_conversation.
api_4_conversation_referenceSame shape as conversation_reference, parented by api_4_conversation.
conversation, api_4_conversationUnchanged by this step: the legacy message and reference payload columns stay in place. The Go schema marks both gorm:"-", so only a probe model can still read them.

Data migration

TableChange
conversation_message, api_4_conversation_messageEach element of the parent row's message JSON array becomes one row, position being its index in the array.
conversation_reference, api_4_conversation_referenceEach entry of the parent row's reference JSON array becomes one row, position being its index in the array. message_position stays NULL: the payload carries no message binding, and only references written together with a message by the current code path set it.

Execution details:

  • Conversations are processed in cursor-ordered batches of 100, each batch in its own transaction, so an interrupted run resumes from the last committed batch.
  • A conversation that already has rows in either child table is left untouched: rows written after the split took effect win over the payload, so a rerun neither duplicates nor clobbers them.
  • Parent tables without message and reference columns are skipped.

On success the step writes v1.0.0-rc1 into the version marker.


v1.0.0-rc2

Converts saved Python v0.27.2 ingestion settings into the component-scoped configuration consumed by Go. After conversion, the existing dataset cleanup removes the retired raptor and graphrag sections.

Data migration

TableChange
knowledgebase, documentMove supported top-level Parser, Chunker, Tokenizer and Extractor settings onto the corresponding pipeline component nodes.
knowledgebaseThe existing cleanup deletes top-level raptor and graphrag sections, preserving other settings.

The conversion covers parser engines and model references, PDF pages, MinerU method/language, TCADP options, spreadsheet column configuration, supported chunk sizes and delimiters, overlap, media context, parent/child settings, filename embedding weight, keywords, questions and metadata. Fields are converted only where the selected component supports them. Settings without a Go consumer are dropped with table/row/field warnings; tags are left untouched, and early Go node formats are normalized by the later v1.0.0.2 step.

Execution details:

  • Component conversion reads each table in id-ordered batches of 256, each batch in its own transaction. Changed rows are updated once by the conversion; unchanged rows retain their original bytes. The existing dataset cleanup performs its own subsequent scan.
  • Each row's parser_id determines the builtin component targets. Parser/Chunker/Tokenizer conversion leaves custom pipeline rows and unsupported parser IDs untouched. Extractor conversion can also target existing Extractor nodes in custom pipeline configs.
  • Existing component values win, except that dataset-level modular metadata transport remains authoritative under the current service contract.
  • SQL NULL, empty strings and JSON null are left untouched. Invalid JSON, unsupported metadata schemas and unsupported engine selections stop migration with table/row context. A failed batch rolls back; earlier committed batches can be rerun safely. See Parser-config conversion failures for recovery instructions.
  • The marker advances only after both table conversions and the existing dataset cleanup succeed. If neither table exists, the marker is left unchanged.

On success the migration records v1.0.0-rc2. Databases already marked v1.0.0-rc2 or later skip conversion. This includes databases marked by the earlier cleanup-only implementation: that marker does not prove their ingestion settings were converted. The later v1.0.0.2 step re-runs the flat-key conversion and normalizes the node-level shapes for those databases.


v1.0.0-rc3.dev1

Permission schema and ownership

The initial permission migration introduced these changes, now retained in the cumulative migratePermissions implementation:

TableSchema and data change
user_tenantMakes invited_by nullable, widens status to varchar(16), and adds unique index idx_user_tenant_user_tenant (user_id, tenant_id).
tenant_model_providerAdds user_id varchar(32) and fills missing or empty owners from tenant_id.
api_tokenAdds nullable user_id varchar(32) and fills missing or empty owners from tenant_id.
user_canvasAdds nullable tenant_id varchar(32) and fills missing or empty tenant IDs from user_id.
tenantAdds invite_enabled with default false and nullable role_id.

Before AutoMigrate creates the membership unique index, migrateUserTenantDuplicates merges duplicate (user_id, tenant_id) memberships. It retains the preferred joined membership using role and timestamp ordering, remaps references in existing group_info, group_member, department_member, and permission tables, and deletes the duplicates. This schema-checked preparation runs independently of the version marker.

The ownership backfills preserve non-empty values. They run after schema creation through migratePermissions, which currently records dev4 after its cumulative work succeeds.


v1.0.0-rc3.dev2

No standalone community schema or data migration for dev2 is defined in this checkout. The community permission target moved directly from dev1 to dev3; dev2 is exercised as an existing database version in the permission migration tests. Upgrading from a dev2 marker runs the cumulative permission migration targeting dev4.

Enterprise permission work is supplied through permissionDataModelsEE and migratePermissionDataEE. This community checkout provides empty/no-op implementations, so it does not define enterprise-only dev2 changes.


v1.0.0-rc3.dev3

Tenant roles and provider ownership

Table or migrationChange
userAdds nullable, indexed role_id for user role assignment.
tenant_model_providerReplaces uniqueness on (provider_name, tenant_id) with idx_tenant_provider_user_unique (provider_name, tenant_id, user_id), allowing different users in one tenant to own providers with the same name.
migratePermissionsInvokes the enterprise permission hook before checking the shared version marker, so a completed community migration cannot suppress enterprise data work. Enterprise implementations can use their own marker through the marker-specific read/write helpers.

Provider index reconciliation is implemented in tenant_model_provider_migration.go. It checks existing indexes by column set, creates the replacement before dropping old two-column unique indexes, and runs independently of the permission version gate. Enterprise system-setting initialization also runs after the migration/server schema setup; it is a no-op in this community checkout.


v1.0.0-rc3.dev4

Legacy tenant invitations

migrateLegacyTenantInvitationStatuses, called by migratePermissions, rewrites user_tenant rows whose role is invite and whose status is joined ('1') or SQL NULL. The resulting role is an empty string and status is invited ('2'). Other memberships remain unchanged.

This step adds no separate schema object. permissionMigrationTargetVersion is now v1.0.0-rc3.dev4; success records that value after the invitation repair and earlier permission backfills finish. A database already at dev4 or later skips the shared permission body, while the enterprise hook still runs before that check.


v1.0.0-rc3.dev5

Tenant default models

Implemented by tenant_default_models_migration.go, with target tenantDefaultModelsTargetVersion (v1.0.0-rc3.dev5).

TableSchema and data change
user_tenantReceives the model-selection columns through AutoMigrate. Missing or empty selections are filled from the matching tenant row only for memberships with role owner or user_id = tenant_id. Existing selections and other members' settings are preserved.
tenantDestructive: drops the old model-selection columns after the transactional copy succeeds.

The columns are llm_id, embd_id, asr_id, img2txt_id, rerank_id, tts_id, ocr_id, and decision_id, together with their corresponding tenant_*_id columns. Copying all columns is transactional; column drops follow and check whether each column still exists, allowing an interrupted run to resume before the marker is advanced.

This version-gated step skips databases already marked dev5 or later. When either required table is absent, there is no copy or drop, but the step still records its target version. It runs before the queue split, which also targets dev5.

Knowledge compilation queues

Introduced in v1.0.0-rc3.dev5 and implemented by migrateKnowledgeCompileQueues in knowledge_compile_queue_migration.go. Each queue is migrated when its source column exists, regardless of mysql_migration.database.version. After both queues finish, the step records knowledgeCompileQueueTargetVersion (v1.0.0-rc3.dev5) in that marker if its current value is older. A newer version is preserved.

TableSchema and data change
knowledge_compile_docs_backlogCreated by AutoMigrate. Each event in knowledge_compile_docs.backlog_doc_ids becomes a row with dataset_id, doc_id, queue_position, event_type, variants, and task_types.
knowledge_compile_docs_inflightCreated by AutoMigrate with the same columns; populated from knowledge_compile_docs.inflight_doc_ids.
knowledge_compile_docsDestructive: drops backlog_doc_ids and inflight_doc_ids, each after its queue has been copied successfully. Dataset scheduling state, leases, progress, and diagnostics remain in this table.

Both new tables have no id, primary key, or unique constraint. A non-unique index on (dataset_id, queue_position) supports ordered reads. Migration preserves duplicate events, array order, and routing metadata; variants and task_types remain JSON arrays stored in longtext columns.

Datasets are read in batches of 100, with each dataset copied in its own transaction. Info-level progress logs identify the queue and batch range, report committed dataset and entry counts and remaining datasets, and record each source-column drop. Entry counts include duplicate events and reset for each queue and migration run. Failed dataset copies roll back and log the dataset and committed progress. Rerunning while services remain stopped replaces copied rows for any source column that still exists, so retries do not duplicate events.

The queue step advances the version marker only after both queue copies and source-column drops succeed. If either queue fails, it leaves the marker at its value on entry; the preceding tenant default-model step may already have recorded dev5. The source-column checks and server startup guard therefore remain authoritative for queue migration completion. Databases whose queues were already split also advance to v1.0.0-rc3.dev5 without copying entries again when their marker is older. The completion log includes the database version and target version.

This migration requires a coordinated service stop:

  1. Stop all services that publish or consume compilation events.
  2. Run ragflow_server --migrate --log-level info to see progress and wait for successful completion of both queues.
  3. Restart the services.

Starting a server while either source column remains fails with knowledge compilation queues need migration. A database version marker at or above the current release does not bypass this requirement. If migration fails, keep services stopped, resolve the reported error, and rerun ragflow_server --migrate.

v1.0.0-rc3.dev6

Locks every pre-existing dataflow-canvas binding to the latest version of that canvas, so already-configured datasets and documents keep parsing against the version that was current when the upgrade ran (design section 5/B4 sticky snapshot). The pipeline_version_id columns it writes are added by the version-less step below; this step only backfills their values and is gated by the dslVersionTargetVersion marker.

Data migration

TableChange
knowledgebase, documentRows whose pipeline_id points at a dataflow_canvas are bound to that canvas latest version (user_canvas_version ordered by create_time DESC). Bindings to a built-in parser id (no backing user_canvas), to a non-dataflow canvas, or to a canvas with no versions are left unbound; a dangling version id is never written.

Execution details:

  • Rows are read in id-ordered batches of 256, each batch in its own transaction, so an interrupted run resumes cleanly.
  • The marker advances to v1.0.0-rc3.dev6 only after both tables are processed. Databases already marked v1.0.0-rc3.dev6 or later skip the backfill.

On success the step writes v1.0.0-rc3.dev6 into the version marker.


v1.0.0-rc3.dev7

Adds a nullable reference LONGTEXT column to conversation_message and api_4_conversation_message. New chat, agent, and channel answers store their reference object directly on their message row.

Run ragflow_server --migrate with writers stopped before starting the new services. The migration copies conversation_reference and api_4_conversation_reference into the corresponding assistant messages. It uses message_position when present; older array entries follow assistant order after the first user message, excluding the prologue. Rows are read in batches of 100 and written in transactions. Reruns preserve reference objects already stored on messages. References duplicated in message metadata are moved into the column and removed from metadata, so generation reads avoid that data.

After both message tables are copied successfully, the migration drops conversation_reference and api_4_conversation_reference. Automatic schema creation no longer recreates these tables. Source-table presence triggers copying and cleanup even if the marker already contains v1.0.0-rc3.dev7 or a newer version; a newer marker is preserved.

References without a surviving assistant message are skipped and counted in a warning. These include references left behind by deleted messages, empty histories, and unsuccessful turns. They are removed with the source tables and are never attached to another answer. Invalid JSON or a copy error stops the migration before either source table is dropped. Resolve the reported issue and rerun it. Completed batches are safe to retry. The marker advances to v1.0.0-rc3.dev7 only after both copies and table drops finish. A failed table drop can be retried even if the other source table has already been dropped.

Session responses return reference within each answer, with no session-level reference array when include_reference=true. Clients must read messages[i].reference (or message[i].reference for agent sessions). include_reference and include_prompt both default to false; omitted fields are excluded from SQL as well as the response. The UI requests the fields it displays. Generation history SQL omits reference, prompt, usage, and elapsed_ms; session detail reads request the fields they display.


v1.0.0.1

Resource-owner user IDs

AutoMigrate adds the user_id columns before migrateResourceUserIDs runs. The version-gated step fills missing or empty user_id values on resource rows so they can be scoped to their effective owner. Existing non-empty user_id values are preserved.

Resource tableOwner source
chat_channel, compilation_template_group, compilation_template, connector, dialog, memory, mcp_server, skill_spacestenant_id
file, knowledgebase, searchcreated_by; if empty, tenant_id
documentcreated_by

Tables that do not exist are skipped. If a listed table exists but its user_id or owner-source column is missing after AutoMigrate, the migration stops with an error. When the source values are empty, the resulting user_id remains empty.

The step updates tables individually rather than in one transaction. If a later table fails, completed updates remain and the version marker is not advanced to v1.0.0.1; rerunning is safe because only missing or empty user_id values are filled. A database already marked v1.0.0.1 or later skips this backfill.


v1.0.0.2

Legacy ingestion component parameters

This step closes the compatibility gap left by v1.0.0-rc2: it rewrites both the top-level flat keys and the pre-modular keys stored inside component nodes. It also re-runs the flat-key conversion so a database whose marker was written by the earlier cleanup-only implementation is converted even though its marker is already v1.0.0-rc2.dev1.

TableChange
knowledgebase, documentNormalize legacy Extractor keys (auto_keywords, keywords_sys_prompt, auto_questions, questions_sys_prompt, auto_tags, tag_file_id, enable_summary, sys_prompt, field_name, metadata_config, fields, and the existing metadata transport keys) into the modular keywords / questions / tags / summary / metadata groups. Normalize TitleChunker rules to levels and TokenChunker delimiter_mode: token_size to delimiter.
user_canvas, user_canvas_versionApply the same normalization to dsl.components[*].obj.params and dsl.graph.nodes[*].data.form, because custom canvas DSLs are the source of truth for custom pipelines.

Conversion rules:

  • Within a component node, existing modular group fields win over flat values. A top-level metadata transport object is authoritative and replaces the node's metadata group; a node metadata group wins over metadata_config, which wins over the flat enable_metadata + field-list form. metadata field lists win over fields.
  • A node's sys_prompt follows its field_name: summary, keywords, and questions keep the prompt in the matching group; metadata, toc, and other values have no prompt target and are dropped with a warning.
  • An explicit enable_summary: false wins over field_name: summary; field_name values other than summary, keywords, and questions have no target in the current Extractor and are dropped with a warning.
  • delimiter_mode: one is not changed: it is a separate legacy alias that dispatches TokenChunker to OneChunker.
  • Custom canvas rows are updated in both the runtime params and the operator form; unrelated components keep their values.

Execution details:

  • Rows are read in id-ordered batches of 256, each batch in its own transaction. Only rows whose params actually change are rewritten.
  • knowledgebase and document rows fail closed: invalid JSON or a malformed supported value stops the step with table/row context. A failed batch rolls back; earlier committed batches can be rerun safely.
  • user_canvas and user_canvas_version rows are user-authored and are not schema-validated on save, so a row whose DSL is not valid JSON or cannot be normalized is logged and skipped instead of blocking the upgrade. The skipped row keeps its legacy shape, which the frontend fallback still reads and re-saves in the modular format; a database write failure still stops the step.
  • The step runs after migrateResourceUserIDs because its target sorts above v1.0.0.1. The marker advances after every table has been processed; a skipped canvas row does not block it. A database already marked v1.0.0.2 or later skips this step.

On success the migration records v1.0.0.2.

Skipped canvas rows

A skipped canvas row is left unchanged and the upgrade still completes, so the service starts normally. The row is not retried on later startups because the marker has already advanced. Search the migration log for either message:

Skipping canvas DSL that is not valid JSON
Skipping canvas DSL the migration cannot normalize

Both carry table, row_id, and the underlying error.

To convert a skipped row:

  1. Open the canvas in the UI and save it. The frontend fallback still reads the legacy keys and writes the modular shape on save.

  2. If the canvas cannot be opened, correct the value named in the warning in user_canvas.dsl / user_canvas_version.dsl, or remove the offending component, then either re-save the canvas or follow step 3 to re-run the step.

  3. To make this step retry the row immediately, reset the shared marker to the version before this step and run ragflow_server --migrate again:

    UPDATE system_settings SET value = 'v1.0.0.1'
    WHERE name = 'mysql_migration.database.version';

    The step is idempotent, so rows it already converted are not changed again.

A skipped user_canvas_version row is a historical snapshot; the live user_canvas row is what the runtime and UI use. Until the row is re-saved or converted, the Go ingestion runtime ignores its legacy flat keys, so the old extractor/chunker settings do not take effect; the frontend fallback still displays them. The skipped row does not block startup.


Version-less steps

These steps are not gated by the version marker. They use schema or data checks to make repeated execution safe. Steps in RunMigrations run during ragflow_server --migrate; ingestion helpers also run during server startup.

Schema changes

TableChange
tenant_llmComposite primary key (tenant_id, llm_factory, llm_name) is replaced by a surrogate auto-increment id; the legacy uniqueness is preserved by a unique index on the old key columns.
task, documentprocess_duation renamed to process_duration.
userUnique index on email.
ingestion_taskUnique index on document_id.
knowledgebaseAdds the VIRTUAL generated column name_ci (LOWER(name) for status='1' rows, NULL otherwise) and the unique index idx_kb_tenant_name_ci (tenant_id, name_ci), so soft-deleted rows never block name reuse.
user_canvasUnique index (user_id, canvas_category, title).
pipeline_operation_logAdds run_count int NULL, pipeline_version_id varchar(32) NULL (the concrete dataflow canvas version a run executed), and the index idx_pipeline_operation_log_document_run.
pipeline_operation_logWidens pipeline_title from varchar(32) to varchar(255) so it can store the user_canvas.title snapshot. Only a narrower column is altered, so rerunning --migrate is a no-op.
tenantWidens name from varchar(100) to varchar(128) to match the shared short-name limit. Only a narrower column is altered, so rerunning --migrate is a no-op.
user_tenant, dialog, knowledgebase, memory, skill_search_configsWidens the tenant_*id model-reference columns (tenant_llm_id, tenant_embd_id, tenant_asr_id, tenant_img2txt_id, tenant_rerank_id, tenant_tts_id, tenant_ocr_id, tenant_decision_id) from varchar(32) to varchar(35) so they can store a gp-<group_id> model group reference. Only a narrower column is altered, so rerunning --migrate is a no-op.
ingestion_taskAdds pipeline_log_id varchar(32) NULL.
ingestion_task_logAdds pipeline_log_id varchar(32) NULL, event_type tinyint NOT NULL DEFAULT 4, and the index idx_ingestion_task_log_pipeline_id.
dialog, tenant_llm, api_token, canvas_template, system_settings, knowledgebaseColumn types converged to the Peewee models, for example dialog.top_k int NOT NULL DEFAULT 1024, tenant_llm.api_key text, api_token.dialog_id varchar(32), canvas_template.title / description nullable longtext, system_settings.value text NOT NULL, knowledgebase.raptor_task_finish_at / mindmap_task_finish_at datetime.
conversation, api_4_conversationIndex idx_<table>_dialog_updated (dialog_id, update_time, id) for conversation list queries.
knowledgebaseAdds pipeline_version_id varchar(32) NULL (dataflow canvas version binding; design section 5/B4).
documentAdds pipeline_version_id varchar(32) NULL (inherits the dataset pinned version at upload; design section 5).
user_canvas_versionAdds alias varchar(255) NULL (free-text version alias for the version list filter; design A2/F1).

migrateColumnWidths performs the width changes above. It reads ColumnTypes().Length() and only issues ALTER COLUMN when the current width is below the target. On MySQL with utf8mb4, widening a VARCHAR column changes its length prefix and can rebuild the table (for example pipeline_title from varchar(32) to varchar(255) rebuilding pipeline_operation_log); run ragflow_server --migrate in a maintenance window and confirm free disk space. The existing pipeline_title index is preserved.

Data changes

TableChange
knowledgebase, dialogparser_config entries stored under the legacy key TokenChunker:SixApplesFall are rewritten to GeneralChunker:SixApplesFall. Values already stored under the new key win; TokenChunker-only fields (delimiter_mode, outputs) are dropped, and a legacy delimiter becomes delimiters. delimiter is discarded when delimiters already exists.
connectorWhen RAGFLOW_CONNECTOR_KEY is set, a plaintext config["credentials"] value is encrypted to an enc:v1: string (AES-256-GCM). Rows that are already encrypted and rows whose config is not valid JSON are left unchanged. Without the key, the step changes nothing.