Sources

Sources organize the data sent to a project. Each source has its own ingest keys and settings, and contains collections of related records.

Create and configure a source

Open project Settings and select Sources.
Click the plus button.
Enter a name and click Create Source.
In the source drawer, open Ingest to copy an ingest endpoint or manage keys.

Creating a source does not guarantee that a key was created. Use Add Key in the Ingest tab if you need one.

Collections and schema detection

Sources begin with no collections. Tailglow chooses one collection for each ingest request: the ?collection= value when it is valid, otherwise the source’s catchall collection. Fields inside the records, including type, never choose the collection. All records in a JSON array or JSONL request use that collection.

Collection names are normalized before they are matched, so ?collection=Charges, ?collection=CHARGES, and ?collection=charges all resolve to the same charges collection. Normalizing trims the value, lowercases it, replaces every character outside a-z, 0-9, _, and - with a hyphen, collapses repeated hyphens, and truncates to 64 characters. A value with no letters or digits left after that is ignored rather than rejected, and the request goes to catchall. The collection’s name field carries its display casing.

A source supports up to 100 collections. New collection names route to catchall after the source reaches that guardrail. A burst of simultaneous requests can briefly create more than 100 collections.

Each collection detail page lists schema versions with their state (candidate, settled, or stale), rows, files, and last-seen time. Use the eye button to inspect a version’s fields and types. Version details shows why it settled and its dates; the drawer also shows whether a field has appeared as null, empty, or absent. Choose Settle version in the drawer when you trust a candidate or stale shape and do not want to wait for it to recur. Settling is one way: a settled version never goes back to candidate. Concepts explains the states. The aggregate Schema Tree remains on the collection page to compare fields across versions.

With Automatic detection, Tailglow groups changing keys when existing mappings can keep working. It leaves a proposal unchanged when a mapping reads inside that path, or when a mapping’s reads cannot be determined. There is no approval task in Automatic mode. With Manual detection, the Schema proposals card shows suggested groups. Its eye button opens the proposed changes, including the remaining shapes and tracked child paths. Choose Group these keys to apply it or Leave unchanged to close it. The drawer warns when an existing mapping may need updating. These changes require sources:write access.

Schema settings

The collection’s Overview includes a Schema recognition summary. Its edit button, or the gear in the page header, opens the settings drawer. Recognition, Settlement, and Limits group the options; choose Save changes to apply edits. Closing the drawer discards unsaved edits. Rules and settings are separate on purpose. Changing a rule re-keys the collection: every version and unclassified shape is re-signed under the new rules, and shapes that now match are merged into one version, which keeps its files and mappings. Before a rule change lands, a confirmation shows how many shapes the collection holds now and how many it would hold under the new rules. Changing a setting re-keys nothing.

Rules:

Grouped paths keep arbitrary children out of schema identity while every raw value is still stored. Enter one path per line, such as /metadata. Escape a literal slash in a key as ~1 and a literal tilde as ~0.
Tracked child paths restore tracking for selected children of an opaque path, such as /metadata/customer_id.
Maximum depth sets how many levels of nesting take part in identity. The default is 5. A mapping can read five levels deep, so raising this past 5 records deeper shapes without making them readable.
Nulls and absent values, Optional keys, and Detection are each Automatic or Manual. Automatic accepts null and absent values at any path a mapping reads, lets a record whose keys are a subset of a version’s keys join that version, and applies a safe opaque-path proposal from the detector on its own. Manual waits for a person at each of those points.

Settings:

Recurrence gap: how long after first sight a shape must be seen again before its version settles. The default is 60 minutes.
Bulk rows: a single batch carrying at least this many rows of a shape settles its version at once. The default is 100, and 0 turns the bulk path off.
Stale after: how long a candidate waits to be seen again before it goes stale. The default is 24 hours.
Settled versions limit, Candidate pool, and Alias pool bound how many settled versions, candidate and stale versions, and cached signatures a collection may hold. Past a bound, new shapes stay unclassified with their signatures kept.
Files per batch: how many collection files one ingest batch may open at once for this collection. A batch with more versions than that is written in more than one pass, so every shape with a version still lands in a file of its own version. Shapes past the candidate pool get the same number of files per batch, the shapes with the most rows first, so they can be linked to a version once the pool has room; the rows of shapes past that budget are kept in the batch’s unclassified file and cannot be linked to a version later.
Unclassified retention: how many days the files of shapes that never settled are kept before they are deleted. That covers unclassified files and the files of stale versions, and a stale version with nothing left under it is deleted with them, together with the pairings views held on it. Leave it empty to keep them forever.

For a version a view cannot map automatically, you can supply a hint, provide a script, or exclude that version from a particular view with a recorded reason.

An ingest key’s recent-use time indicates that requests reached the source. It does not mean every view and chart has finished processing them.

Deleting a source

Use the hold-to-delete control in the source drawer to schedule deletion. The source remains visible with a line-through name and a Deleting chip. Hover the chip or open the source drawer to see how long remains before permanent removal. Tailglow schedules removal for seven days later.

Before that date, open the source drawer and click Restore Source to cancel deletion. The source keeps accepting data for the whole window, so a restored source carries on with nothing missing.

The same drawer offers Delete Permanently if you would rather not wait. It asks you to type the source name, destroys every collection and record under it immediately, and cannot be undone.

Permanent removal deletes the source, its collections and data, and any views built from those collections. Existing results from joins that used one of those views remain. If the removed view was the join’s main input, the join stops receiving new records; if it was a lookup, new records are no longer enriched by it. Reconfigure or remove affected joins before permanent removal.