Manage data retention with TTL policies

This page describes how to use the Google Cloud console and the Google Cloud CLI to configure time to live (TTL) policies. Before you read this page, you should understand the Cloud Firestore data model.

Time to live overview

Use TTL policies to automatically remove stale data from your databases. A TTL policy designates a given field as the expiration time for documents in a given collection group. With TTL, you can decrease storage costs by cleaning out obsolete data. Data is typically deleted within 24 hours after its expiration date.

Pricing

TTL delete operations count towards your document delete costs. For pricing of delete operations, see Cloud Firestore pricing.

Limits and constraints

• Only one field per collection group can be marked as a TTL field.
• A total of 200 field level configurations are allowed. One field configuration can contain multiple configurations for the same field. For example, a single-field indexing exemption and a TTL policy on the same field count as one field configuration towards the limit.
• For Firestore in Datastore mode customers, TTL cannot be used with a concurrency mode of Optimistic With Entity Groups. Consider changing the concurrency mode to the Optimistic concurrency mode.

TTL deletion

Note the following key behaviors of TTL-driven deletion:

• Deletion through TTL is not an instantaneous process. Expired documents continue to appear in queries and lookup requests until the TTL process actually deletes them. TTL trades deletion timeliness for the benefit of reduced total cost of ownership for deletions. Data is typically deleted within 24 hours after its expiration date.

• Deleting a document through TTL does not delete subcollections under that document.

• Applying a TTL policy on an existing collection group results in a bulk deletion of all expired data according to the new TTL policy. Note that this bulk deletion is also not instantaneous and depends on how much data exists for that collection group.

• If a document has an expiration time in the past and you add a new TTL policy to the collection, the document will be deleted within 24 hours of when the TTL policy finishes setup and becomes active.

• TTL does not necessarily delete documents in the same order as their expiration timestamps.

• Deletions are not done transactionally. Documents with the same expiration time are not necessarily deleted at the same time. If you require this behavior, perform the deletions using a client library.

• Cloud Firestore will always honor the latest TTL field to determine the expiration. For example, if an expired but not-yet-deleted document has its TTL field updated to a later date, the document will not be expired and the new date will be used.

• TTL is designed to minimize impact on other database activities. Deletions driven by TTL are treated with a lower priority. Other strategies are also in place to smooth out traffic spikes from TTL-driven deletes.

• Deletion through TTL calls all active snapshot listeners and triggers Cloud Functions Cloud Firestore triggers.

TTL fields and indexes

A TTL field can be indexed or unindexed. However, because a TTL field is a timestamp, indexing the field can affect performance at higher traffic rates. Indexing a timestamp field can create hotspots which is against best practices. Hotspots are high read, write, and delete rates to a narrow document range.

By default, Cloud Firestore creates a single-field index for all fields. You can create a single-field index exemption to disable indexes on a TTL field.

Permissions

The principal configuring a TTL policy requires the following permission in the project:

• Viewing TTL policies requires the datastore.indexes.list and datastore.indexes.get permissions.
• Modifying TTL policies requires the datastore.indexes.update permission.
• Checking the status of TTL operations requires datastore.operations.list and datastore.operations.get.

For roles that assign these permissions, see Cloud Firestore Identity and Access Management roles.

Before you begin

Before you use the gcloud CLI to manage TTL policies, use the gcloud components update command to update components to the latest available version:

gcloud components update

Create a TTL policy

When you create a TTL policy, you designate a document field as the expiration time for documents in a collection group.

TTL uses a specified field to identify documents that are eligible for deletion. This TTL field must be of the Date and time type. You can select a field that already exists or you can designate a field that you plan to add later.

Consider the following before you set the TTL field value:

• The TTL field value can be a time in the future, now, or in the past. If the value is a time in the past, the document is immediately eligible for deletion. For example, you might create a TTL policy with the field expireAt, which you then add to existing documents.

• Using any other data type or not setting the TTL field value will disable the TTL for the individual document.

To create a TTL policy, follow these steps:

 gcloud firestore fields ttls update
  ttl_field --collection-group=collection_group_name
  --enable-ttl 

TTL policy enablement duration

Even on an empty database, it can take ten minutes or more to enable a TTL policy. Once you start an operation, closing the terminal does not cancel the operation.

View TTL policies

To view TTL policies and their statuses, follow these steps:

   gcloud firestore fields ttls list
   

   gcloud firestore fields ttls list  --collection-group=collection_group_name
   

View operation details

You can use the gcloud CLI to view more details about a TTL policy that is in the CREATING state.

Use the operations list command to see all running and recently completed operations:

gcloud firestore operations list

The response includes an estimate of the operation's progress.

Disable a TTL policy

To disable a TTL policy, follow these steps:

   gcloud firestore fields ttls update ttl_field --collection-group=collection_group_name --disable-ttl
   

Monitor TTL deletions

You can use Cloud Monitoring to view metrics about TTL-driven deletions. Cloud Firestore provides the following metrics for TTL:

To set up a dashboard with Cloud Firestore metrics, see manage custom dashboard and add dashboard widgets.