Collections

Collection Model

Fields

Field

Type

Description
object
"collection"
id
string Unique identifier, prefixed with col_.
source_id
string
source_name
string | null
project_id
string
slug
string Permanent identifier used in endpoints. Cannot be changed after creation.
name
string
storage_gb
number Data stored in this collection's files, in gigabytes of uncompressed data. Recalculated hourly.
schema_count
number How many schema versions this collection has, in every state.
views_count
number How many views read from this collection.
drains_count
number How many drains export from this collection.
schema_opaque_paths
string[][] Object paths whose contents do not take part in schema identity, one key segment per entry. Records that differ only inside an opaque path share one schema version. Changing this re-keys the collection.
schema_tracked_paths
string[][] Children of opaque paths that stay part of schema identity. Changing this re-keys the collection.
schema_max_depth
number How many levels of nesting take part in schema identity. Changing this re-keys the collection.
schema_nulls
auto accepts null and absent values where a mapping reads; manual waits for a person per view.
schema_optional_keys
auto lets a record with a subset of a version's keys join it; manual treats absence as a new shape.
schema_detector
auto applies a safe opaque-path proposal on its own; manual holds every proposal for a person.
schema_recurrence_gap_ms
number How long after first sight a shape must recur before its version settles, in milliseconds.
schema_bulk_rows
number A single batch carrying this many rows of a shape settles its version at once; 0 disables it.
schema_stale_after_ms
number How long a candidate version waits to recur before it goes stale, in milliseconds.
schema_identity_limit
number How many settled versions the collection may hold; past it, new shapes stay unclassified.
schema_candidate_pool
number How many candidate and stale versions the collection may hold; past it, new shapes stay unclassified.
schema_alias_pool
number How many exact signature hashes the collection caches for lookup.
schema_writer_limit
number How many collection files one ingest batch may open for this collection. Shapes past the candidate pool get the same number of files per batch, the shapes with the most rows first; the rows of shapes past that budget stay unclassified and cannot be linked to a version later.
schema_unclassified_retention_days
number | null How many days rows without a settled version are kept; null keeps them forever.
deleted_at
| null When the collection is scheduled to be deleted; null when it is not.
created_at
updated_at

Collection Schema Report

Fields

Field

Type

Description
object
"collection_schema_report"
collection_id
string
rules
{ opaque_paths: string[][]; tracked_paths: string[][]; max_depth: number; } The rules the report reads the collection's shapes under: the collection's own for the retrieve, the rules sent for a preview.
shapes
{ before: number; after: number; } How many shapes the collection holds under its current rules (before) and under the report's rules (after). Equal unless the report previews a rule change; the difference is what a re-key would merge.
proposals
[] Where the detector sees keys being invented, with what applying each proposal would do.
keys_look_like_data
boolean Whether every shape's top-level keys are its own, such as records keyed by a date. Those keys are values, and no opaque path can bring the shapes together.
unclassified
{ shapes: number; rows: number; } Shapes past the collection's pools, stored with their signature but no version, and the rows they carry. Nothing reads them until the pool frees or a rule brings them under a version.

Collection Schema Proposal

Fields

Field

Type

Description
object
"collection_schema_proposal"
path
string[] The path the detector proposes opaque, as key segments.
tracked
string[] Children of the path every shape carries with one type, which stay in schema identity as tracked paths when the proposal is applied.
content_children
number How many children of the path read as content: keys that keep being invented.
read_by_mapping
boolean Whether a bound mapping reads inside the path. Applying the proposal would leave that mapping unverified, so the detector never applies such a proposal on its own; a person can.
shapes_after
number How many shapes the collection would hold with this proposal applied.

List Collections

Endpoint

Retrieve a list of collections for a project.

sources:read
GET
/v1/projects/:project_id/collections

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.

Query Parameters

Field

Type

Description
order_by
string Field used to order the collections. Defaults to "name". Accepted values: "created_at","name","updated_at".
source_id
string Filter to collections under a single source.
project_id
string Filter to collections in a single project.
deleted_at
NullableDateFilter Filter by scheduled deletion date. Use null for collections that are not scheduled for deletion, not:null for collections that are.
limit
number Defaults to 25. Minimum: 1. Maximum: 200.
after
string Cursor from pagination.next_cursor of a previous response. Returns the resources after that page.
before
string Cursor from pagination.prev_cursor of a previous response. Returns the resources before that page.
sort
string Defaults to "asc". Accepted values: "asc","desc".

Comments

  • after and before are mutually exclusive.

Response

200 
{
  message: string;
  data: Collection[];
  status: 200;
  error: null;
  pagination: Pagination;
  endpoint: string;
}

Retrieve Collection

Endpoint

Retrieve a single collection.

sources:read
GET
/v1/projects/:project_id/collections/:collection_id

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Response

200 
{
  message: string;
  data: Collection;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

Create Collection

Endpoint

Create a collection under a source.

sources:write
POST
/v1/projects/:project_id/collections

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.

Request Body

Field

Type

Requirement

Description
name
string
Required
Display name for the collection. Minimum length: 1. Maximum length: 128.
slug
string
Optional
Permanent identifier used in ingest URLs. Derived from name when omitted. Minimum length: 1. Maximum length: 64.
source_id
string
Required
Source the collection is created under.

Comments

  • source_id is required.
  • slug is permanent. It appears in the ingest URL, so pick it deliberately.

Response

201 
{
  message: string;
  data: Collection;
  status: 201;
  error: null;
  pagination: null;
  endpoint: string;
}

Update Collection

Endpoint

Update a collection.

sources:write
POST
/v1/projects/:project_id/collections/:collection_id

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Request Body

Field

Type

Requirement

Description
name
string
Optional
Display name for the collection. Minimum length: 1. Maximum length: 128.
expected_updated_at
Optional
Reject this update if the collection changed since this inspected update time.
schema_opaque_paths
string[][]
Optional
Object paths whose contents do not take part in schema identity, each written as its key segments (["metadata", "id"]). A path that names an array covers its elements and their children. Records that differ only inside an opaque path share one schema version; the raw values are still stored. Changing this re-keys the collection.
schema_tracked_paths
string[][]
Optional
Children of opaque paths that stay part of schema identity, written like schema_opaque_paths. Changing this re-keys the collection.
schema_max_depth
integer
Optional
How many levels of nesting take part in schema identity. Changing this re-keys the collection. Minimum: 1. Maximum: 10.
schema_nulls
string
Optional
auto accepts null and absent values at any path a mapping reads; manual waits for a person to approve each nullable input per view. Accepted values: "auto","manual".
schema_optional_keys
string
Optional
auto lets a record whose keys are a subset of a version's keys join that version; manual treats every absent key as a different shape. Accepted values: "auto","manual".
schema_detector
string
Optional
auto applies a safe opaque-path proposal from the detector on its own; manual holds every proposal for a person. Accepted values: "auto","manual".
schema_recurrence_gap_ms
integer
Optional
How long after first sight a shape must be seen again before its version settles, in milliseconds. Minimum: 60000. Maximum: 604800000.
schema_bulk_rows
integer
Optional
A single batch carrying at least this many rows of a shape settles its version immediately. 0 disables the bulk path. Minimum: 0. Maximum: 1000000.
schema_stale_after_ms
integer
Optional
How long a candidate version waits to be seen again before it goes stale, in milliseconds. Minimum: 3600000. Maximum: 7776000000.
schema_identity_limit
integer
Optional
How many settled schema versions the collection may hold. Past it, new shapes stay unclassified with their signatures kept. Minimum: 1. Maximum: 4096.
schema_candidate_pool
integer
Optional
How many candidate and stale versions the collection may hold. Past it, new shapes stay unclassified. Minimum: 1. Maximum: 8192.
schema_alias_pool
integer
Optional
How many exact signature hashes the collection remembers for fast lookup. Past it, a new hash is still resolved but not cached. Minimum: 16. Maximum: 100000.
schema_writer_limit
integer
Optional
How many collection files one ingest batch may open for this collection. Shapes past the candidate pool get the same number of files per batch, the shapes with the most rows first; the rows of shapes past that budget stay unclassified and cannot be linked to a version later. Minimum: 1. Maximum: 256.
schema_unclassified_retention_days
integer | null
Optional
How many days rows without a settled schema version are kept before they are deleted. null keeps them forever.
deleted_at
null
Optional
Send null to restore a collection that is scheduled for deletion. The deletion date itself is set by the delete endpoint and cannot be written here.

Comments

  • slug cannot be changed after creation.
  • Send deleted_at as null to restore a collection that is scheduled for deletion.

Response

200 
{
  message: string;
  data: Collection;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

Delete Collection

Endpoint

Schedule a collection for deletion.

sources:delete
DELETE
/v1/projects/:project_id/collections/:collection_id

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Comments

  • The collection is hidden immediately and permanently removed 7 days later. Restore it before then by updating it with deleted_at set to null.

Response

200 
{
  message: string;
  data: Collection;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

Force Delete Collection

Endpoint

Delete a collection permanently, without waiting out its restore window.

sources:delete
DELETE
/v1/projects/:project_id/collections/:collection_id/force

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Comments

  • Works on a live collection as well as one already scheduled for deletion.
  • Every record, schema version and file in the collection is destroyed immediately.

Response

200 
{
  message: string;
  data: null;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

List Collection Schemas

Endpoint

Retrieve the schema versions for a collection.

sources:read
GET
/v1/projects/:project_id/collections/:collection_id/schemas

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Query Parameters

Field

Type

Description
limit
number Maximum number of items to return. Defaults to 25. Minimum: 1. Maximum: 200.
after
string Cursor from pagination.next_cursor of a previous response. Returns the resources after that page.
before
string Cursor from pagination.prev_cursor of a previous response. Returns the resources before that page.
sort
string Sort direction for the result set. Defaults to "desc". Accepted values: "asc","desc".
state
string Return only schema versions in this state. Accepted values: "candidate","settled","stale".

Comments

  • state is candidate from first sight, settled once the shape recurs or arrives in bulk, and stale when a candidate is not seen again. Only a settled version is offered to views for authoring.
  • after and before are mutually exclusive.

Response

200 
{
  message: string;
  data: CollectionSchemaVersion[];
  status: 200;
  error: null;
  pagination: Pagination;
  endpoint: string;
}

Update Collection Schema

Endpoint

Settle a schema version by hand, so views can author against it before its shape recurs.

sources:write
POST
/v1/projects/:project_id/collections/:collection_id/schemas/:schema_version_id

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.
schema_version_id
string Unique identifier of the schema.

Request Body

Field

Type

Requirement

Description
state
string
Required
The only state a person sets. settled trusts the version for view authoring now, without waiting for its shape to recur; a settled version never goes back. Accepted values: "settled".

Comments

  • state accepts only settled; a settled version never returns to candidate or stale.
  • A version that is already settled is left as it is and responds with request_no_update.

Response

200 
{
  message: string;
  data: CollectionSchemaVersion;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

Retrieve Collection Schema Report

Endpoint

Retrieve the detector's view of a collection: the opaque paths it proposes, whether the top-level keys are data, and how many shapes sit unclassified past the collection's pools.

sources:read
GET
/v1/projects/:project_id/collections/:collection_id/schema_report

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Comments

  • A proposal a bound mapping reads inside is reported and never applied on its own, whatever the detector switch; apply it by adding its path to the collection's schema_opaque_paths and its tracked children to schema_tracked_paths.
  • shapes.before and shapes.after are equal here; the preview is where they differ.

Response

200 
{
  message: string;
  data: CollectionSchemaReport;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

Preview Collection Schema Rules

Endpoint

Preview a rule change: the schema report the collection would read under the rules sent, with how many shapes it holds now and how many it would hold. Nothing is changed.

sources:read
POST
/v1/projects/:project_id/collections/:collection_id/schema_preview

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Request Body

Field

Type

Requirement

Description
schema_opaque_paths
string[][]
Optional
Opaque paths to preview, written like the collection's schema_opaque_paths. Omitted, the collection's current opaque paths apply.
schema_tracked_paths
string[][]
Optional
Tracked paths to preview, written like the collection's schema_tracked_paths. Omitted, the collection's current tracked paths apply.
schema_max_depth
integer
Optional
Max depth to preview. Omitted, the collection's current max depth applies. Minimum: 1. Maximum: 10.

Comments

  • A rule left out of the body keeps the collection's current value.
  • shapes.before counts the shapes under the collection's current rules and shapes.after under the rules sent; the difference is what updating the collection with those rules would merge.

Response

200 
{
  message: string;
  data: CollectionSchemaReport;
  status: 200;
  error: null;
  pagination: null;
  endpoint: string;
}

List Collection Documents

Endpoint

Retrieve raw documents from a collection.

sources:read
GET
/v1/projects/:project_id/collections/:collection_id/docs

Path Parameters

Field

Type

Description
project_id
string Unique identifier of the project.
collection_id
string Unique identifier of the collection.

Comments

  • Documents are returned exactly as they were sent, minus Tailglow's own reserved envelope.
  • after and before are mutually exclusive.

Response

200 
{
  message: string;
  data: CollectionDoc[];
  status: 200;
  error: null;
  pagination: Pagination;
  endpoint: string;
}

Referenced Types

NullableDateFilter

Nullable date filters accept everything a date filter accepts, plus null to match records where the field is unset and not:null to match records where it is set.

ISODateString

An ISO 8601 date-time string returned at the JSON API boundary.

SchemaRuleMode

auto
manual