Concepts

Introduction

Tailglow separates the ingest boundary from the data boundary. The core concepts are:

• Sources: Authentication and ingest configuration for a project.
• Collections: Logical streams of records within a source. Collections own schema versions, files, and raw records.
• Views: Transformed data derived from a collection or produced by joining views.
• Metrics: Aggregated measurements and charts built from views.

Sources and collections

A source is the place you configure ingest keys. Its Ingest tab displays the project’s default endpoint after one has been provisioned and any available team ingest domains. If neither exists, the tab says No ingest endpoints configured. The project and team domains own those routing endpoints; the ingest key selects the source. A new source starts with no collections.

Tailglow chooses one collection for each ingest request. A valid ?collection=charges query parameter picks it, and a request without one goes to the source’s catchall collection. Fields inside the records, including type, never choose the collection. Every record in a JSON array or JSONL request goes to that collection.

A source supports up to 100 collections. New collection names route to catchall after the source reaches that guardrail. A burst of simultaneous requests can briefly create more than 100 collections.

Records

Records are the events you send to an ingest endpoint. Tailglow accepts JSON, JSONL, CSV, TSV, PSV, SSV, and plain text. Raw collection records can be sampled as soon as they arrive, whatever the state of their schema version.

JSON records need no fixed wrapper. Tailglow normalizes tabular and text input into JSON records, renames unsafe property keys, adds delivery metadata, and replaces embedded image data with artifact references.

Schema versions

Every record that reaches a collection is read for its shape: the paths it carries and the type at each path, down to the collection’s depth rule. The first time a shape is seen, the collection registers a schema version for it and stamps every file holding that shape with the version. A version’s number is permanent. Its canonical shape can still refine in place: a path first seen blank learns its type, and the paths a wider record carries are added once that shape has come back or arrived in bulk, so a single stray key never changes a version. A rule change can merge two versions into one, and the merged version’s files move to the survivor. A record that is not a JSON object, or an empty one, has no shape to register: it is stored and kept, but it never joins a schema version and no view reads it.

A version is in one of three states:

• Candidate: the shape has been seen once. Its records are stored and served immediately by any view mapping that already fits them, but no mapping is written for it yet.
• Settled: the shape was seen again after the collection’s recurrence gap, arrived in one batch with at least the bulk row count, or a person settled it. Only settled versions are offered to views for authoring.
• Stale: a candidate that was not seen again within the stale window. Its rows stay stored but unclassified. A stale shape that returns settles.

Each view decides separately whether an existing TGL script can read a version. Changes under fields a script never reads reuse the script at once. A change from name: string to name: {first, last} needs a new decision for a view that reads the name, while a view that only reads run_id continues.

Nulls and empty values never make a new version: a version first seen with user_id: null refines in place once strings arrive. Absent keys follow the collection’s optional keys switch. On automatic, a record whose keys are a subset of a version’s keys joins that version. On manual, absence is structural, and a record missing a key is a distinct version. A mapping written while the path had no values is flagged on the view with the refined paths, and re-authoring it clears the notice. See Schema settings for the rules and settings a collection carries.

Views

Views reshape data with TGL. In the Views page, click New View and choose one of the available creation modes:

• Collection: Choose a Source Collection, optionally define an output schema and a generation hint, then click Create View. Tailglow generates transforms for the collection’s settled schema versions.
• View: This is a join builder, not a simple parent-derived view. Choose a base view, add one or more lookup views, and select the fields that join them.

The view’s Overview shows how source versions flow through scripts into the output schema and linked metrics, drains, or views. Narrow panels use a connected vertical layout. More than two scripts become a counted group; open it for the script list, or open a source group to choose an individual version. Failed and processing scripts remain visible in the group’s status. Counts describe the loaded data; Version coverage lets you load more versions.

The saved Generation hint is shown directly below the diagram on desktop and mobile. Change it in the view’s Settings. If none is saved, the diagram says No generation hint added. Users who can edit the view can click the + beside it to open Settings and add a hint.

You can inspect transformed data on a view’s Records tab.

Metrics

Metrics aggregate data from a view. Choose a timestamp field, optional grouping, filters, and value or unique-count fields. Use metrics to chart counts, numeric values, and grouped trends, then attach monitors when you need alerts. Events with overridden timestamps that land far in the past are held for review rather than charted silently; Late data explains the window.

Summary

Sources authenticate and configure ingest. Collections own the raw data and its schema versions. Views transform or join collection data one schema version at a time. Metrics aggregate view output for analysis and monitoring.