Upsert

Description

upsert() is a Data manipulation language (DML) stage that updates existing documents or inserts new documents if they don't exist. This stage can be attached to the end of a query.

The upsert() stage can take different arguments to determine the destination collection, document ID, and optional field transformations:

• upsert(collectionName, documentIdExpression, ...transformations): Upserts documents into the specified collection, using the evaluated expression to determine the document ID, and applies optional transformations.
• upsert(collectionName, ...transformations): Upserts documents into the specified collection, requiring a __name__ field in the input document to determine the document ID, and applies optional transformations.
• upsert(...transformations): Upserts documents, requiring a __name__ field in the input document to determine the destination collection and document ID, and applies optional transformations.

Examples

Migrate active users

The following example queries the users collection where age >= 25, adds a processed field, and upserts into new_users using the name field as the document ID:

Behavior

The following sections describe the execution behavior, responses, transformations, and restrictions for the upsert() stage.

Upsert in place with field transformations

The following example updates existing documents in the users collection in place by modifying or adding fields during the upsert:

Upsert into a different collection with explicit ID

The following example reads from source_coll and upserts documents into target_coll using an explicit document ID "new_doc_id" and applying a field transformation:

Upsert into another collection using existing document ID (__name__)

The following example reads documents from source_coll and upserts them into target_coll using each document's existing document ID (__name__):

Upsert with document references (__name__)

The following example uses document references assigned to the __name__ field to specify destination collection and document ID:

Upsert with auto-generated document IDs

The following example upserts literals into the users collection with auto-generated document IDs and field transformations:

Upsert into a Subcollection

The upsert() stage can be used to insert or update documents directly into a subcollection using a full path.

Upsert from Advanced Queries (Search and Find Nearest)

The upsert() stage can be used with advanced query operations like search or findNearest to write the matching documents into an output collection or update them in place.

Upsert from Aggregation Subqueries

The upsert() stage can be used with aggregation subqueries and scalar functions to update documents based on dynamically computed values, such as identifying if a user has the top score. This pattern must be executed within a user transaction.

Response

The upsert() stage always emits a single document like { documents_modified: 3 } describing how many documents were inserted or updated.

Final Stage

The upsert() stage must come at the end of a pipeline, no further stages can be provided.

Transaction Required

The upsert() stage requires a transaction. It must be run within an active read-write transaction, or with the atomic (Node.js) or auto_commit option set to true when starting a new transaction.

Response

The response includes a summary of the number of documents modified. For example, the following response confirms that the pipeline modified three documents:

{documents_modified: 3L}

Limitations

• DML stages can now be run within transactions. The insert() and upsert() stages need to be run within a transaction in most cases.

• When executing a pipeline with DML stages within a new transaction, you can use the atomic option to automatically run within a transaction and commit upon successful execution of the pipeline.

• When running within a transaction, there are no restrictions on other stages (such as where, addFields, union, findNearest, and aggregate subqueries). Transactional DML operations are bounded by Firestore Enterprise edition transaction limits. Because transactions don't allow reads after the first write, you cannot call a DML pipeline multiple times within a single transaction or read documents after a DML pipeline executes. Attempting to execute a read after a DML write fails with a FAILED_PRECONDITION error: Cannot have reads after the first write in a Firestore transaction.

• When running outside of a transaction:

  • Each document is updated independently of each other, and writes are not atomic across documents. The pipeline halts on the first error, allowing for partial success to occur.
  • Advanced multi-query stages (aggregate, distinct, unnest, find_nearest, union, joins, and subqueries) are not supported.

• Under Firebase Security Rules, each batch of documents re-evaluates rules and executes lookups. If rules use get() or exists(), the global 20-lookup limit is reached after writing approximately 200 documents, and the operation fails with PERMISSION_DENIED. Use Custom Claims instead of get() / exists() lookups to avoid this limit.

• If the stage preceding the DML stage produces multiple documents with the same __name__ or evaluates to the same destination document ID, each mutation is processed sequentially:

  • For upsert(...) and update(...): Multiple mutations targeting the same document follow "last one wins" semantics, where the last mutation processed determines the final state of the document.
  • For delete(...): The first attempt deletes the document, and subsequent attempts for the same document are no-ops.
  • For insert(...): The first attempt creates the document, and subsequent attempts targeting the same document ID fail with an ALREADY_EXISTS error.

Transactional Execution

DML operations can be executed as part of a read-write transaction by either providing an explicit transaction to the pipeline execution call or by using the atomic option.

Firestore transactions require all reads to execute before any writes. Because a DML pipeline executes writes, you cannot perform reads (such as document lookups or query reads) after running a DML pipeline in the same transaction. For example, you cannot call a DML pipeline multiple times in a single transaction. Attempting to read after a DML pipeline executes fails with a FAILED_PRECONDITION error: Cannot have reads after the first write in a Firestore transaction.

Using an explicit transaction

Using the atomic option