Building analytics in week two
In the second week of Giant Context, before there was much of a product to measure, I built the analytics. Not a feature. A primitive, in the same category as the database and the permission model, and for the same reason: everything else gets to assume it.
A platform needs to know what happened. Which page a visitor loaded, which request was slow, which call spent tokens, which thing threw. Those look like four separate systems and they are one shape. Build that shape once, early, and each of them becomes a question you ask of a record you already keep, rather than a subsystem you have to go and build.
The easy path is to let a customer wire up Google Analytics and call it handled. A rented dashboard can drive a website. It cannot drive a platform, because a platform has to bill from the same numbers, alert on them, and feed them back into what it makes next, and none of that is something you do against a tool a person signs into. It is something you do against a table.
One event contract
So there is one event pipeline. A single contract, one shape that anything in the system emits, and one place it lands. A visitor loading a customer's page is an event. So is an API request returning, a model call spending tokens, a page publishing, an account signing up, an error thrown in a browser three time zones away.
One table holds all of it, fifty-five columns wide, in BigQuery. There is no Postgres analytics table, no message queue, no outbox pattern. Writes are streaming inserts straight into the table. One destination means nothing to keep in sync, and one contract means nothing to translate between.
analytics.events, abridged
event_id STRING timestamp TIMESTAMP category STRINGaction STRING tokens INTEGER user_id STRINGtrace_id STRING session_id STRING organization_id STRINGproject_id STRING app_id STRING app_type STRINGresource STRING endpoint STRING http_method STRINGstatus_code INTEGER duration_ms FLOAT platform STRINGdevice_type STRING browser STRING os STRINGuser_agent STRING ip_address STRING country_code STRINGcity STRING latitude FLOAT longitude FLOATreferrer STRING utm_source STRING utm_medium STRINGproperties STRING ingested_at TIMESTAMP
-- 55 columns in totalFour kinds of thing emit into it, and only one of them is a browser.
Everything lands in one table
PRODUCERS INGEST STORE
public site (SSR) ─┐ page render │ render errors │ │ POST /analytics/eventsconsole (browser) ─┤ ─────────────────────► enrich page views │ batched, keepalive IP, geo, feature + clicks │ device │ │services ─┘ ▼ router, brain, ai analytics.events ▲API request hook ──────────────────────────────────────┘ in-process, no network hopThe API's own request events skip the network entirely. A response hook builds the event and writes it in process, because the service doing the writing is the service that would have received the post. Everything else goes through the endpoint.
That endpoint answers 204 to everything. Not 202, not a body, not an error on a bad batch. A caller cannot tell a successful write from a dropped one, which is intended: the alternative is a page render or an API response that got slower or failed on account of a measurement. Losing an event costs nothing. Slowing a page costs a visitor.
Before writing, the receiver enriches. The client never sends an IP or a location and is never trusted with either. The receiver reads the request address, then takes country, region, city and coordinates from the load balancer's own headers rather than doing a geo lookup, which means location costs nothing and arrives with the request. Browser, operating system and device are derived from the user agent, but only where the event did not already supply them.
Anything marked as an error is written twice, once to the main table and once to an errors table with its stack trace parsed and a grouping signature computed from the error code, file, line and function. Same event, one extra write, and duplicates of the same fault collapse into one thing to look at rather than a thousand.
Three decisions that look arbitrary
Three decisions in there look arbitrary until you hit the thing that caused them.
The browser tracker batches, ten events or five seconds, and flushes on the page being hidden as well as on unload, because a tab switched away from is far more common than a tab closed. It sends with the browser's beacon only when the endpoint is same-origin, and falls back to a keepalive fetch otherwise. Beacons go out with credentials disabled, so cross-origin they are accepted by the browser and dropped in silence. Nothing errors. The events simply never arrive, and you find out by noticing a number is too low.
Every environment shares one dataset, which is not what a textbook would tell you to do. The consequence is that a developer running the platform locally would write into the same table as production. So local traffic is dropped at the receiving endpoint, on the way in, rather than filtered out at read time. That sounds backwards until you consider who would do the filtering: a browser cannot be trusted to say which environment it belongs to, and a query-time filter is a rule every future query has to remember. Dropping on arrival is the only place the decision gets made once.
And the platform's own traffic is separated from real visitors by a string comparison. Server-to-server calls go out through Node, whose default user agent is the word node, and it is stored exactly as received. Every website query carries a condition excluding it. That works, and it is a decision made by the shape of the data rather than by design, which means it holds until something internal starts sending a different agent.
A page view with no script and no cookie
Most website analytics is a script tag. A visitor loads the page, the browser fetches a third-party bundle, that bundle sets a cookie and posts back. It costs the visitor a request and it costs the site owner a consent banner.
A page view here is a server component. It sits in the rendered tree, reads the organization, project, app and path from the request headers, and emits one event per render. Nothing ships to the browser. There is no script to block, nothing to slow the paint, and it works with JavaScript switched off.
It also sets no cookie, because it has nothing to identify. The console's browser tracker keeps a session id in session storage, which dies with the tab and never reaches disk. Google Tag Manager, which a customer can switch on if they want it, is the only capture on the whole platform gated behind consent, because it is the only one that needs to be.
What it cannot see is anything the browser learns after the fact. Time on page, scroll depth, whether a visitor clicked a button rather than merely loaded the page holding it. Counting arrivals accurately came first, and engagement is a harder problem this design does not solve.
A chart is a query
Because there is only one table, a chart is a query rather than a subsystem. The traffic graph on a customer's dashboard is this, with the filters bound as parameters.
Website traffic over time
SELECT TIMESTAMP_TRUNC(timestamp, DAY) AS date, COUNT(*) AS countFROM analytics.eventsWHERE <filters> AND resource = 'apps/website' AND user_agent != 'node'GROUP BY dateORDER BY date ASCEvery chart runs live. There is no results table, no materialized view, no server-side cache; a chart request builds its query, runs it, and returns. The only caching is in the browser, sixty seconds of it, which means a dashboard left open is not quietly re-billing me for the same question and a chart labelled live is refreshed by a page load rather than a poll.
Countries, cities, pages and devices are the same query grouped by a different column. Every chart on the platform is a config declaring its filters, its metric and its shape, compiled into parameterized SQL against that one table, and served by a single endpoint that takes the name of a dimension. Adding a chart is adding a config, not a feature.
One gap is worse than the others. The table has columns for the referring page and for every UTM parameter, and nothing fills them. The page event reads host, path and locale from the request and never touches the referring header or the query string. So a system built to learn which marketing worked cannot presently say where a visitor came from, and those columns have been empty since the first insert.
The same table produces the invoice
There is a column on every event called tokens, and on most rows it is zero. On the rows where it is not, it is the cost of goods.
My unit cost is tokens spent calling a model. Every generation the product performs emits an event like everything else, and that event carries what it cost. So the invoice is a query against the same table the visitor counts come from, summed by organization, with the categories that are not billable left out.
What a customer owes
SELECT organization_id, project_id, SUM(tokens) AS total_tokensFROM analytics.eventsWHERE timestamp >= @startDate AND timestamp < @endDate AND organization_id IS NOT NULL AND category NOT IN UNNEST(@excludedCategories)GROUP BY organization_id, project_idErrors, warnings, system work and authentication are excluded, because a customer should not pay for the platform's own failures or its housekeeping. A daily job rolls that up into a snapshot in Postgres so billing is not re-querying months of history, but the snapshot is derived. The event table remains the only record.
I did not anticipate this when I built the contract. It was built so the AI could see results. But a page view, a trace, an error and a line on an invoice turn out to be the same shape: something happened, here is who to, here is where, here is what it cost. Three systems I would otherwise have built separately are one query each against a table that already existed.
What owning the data buys
Analytics is not a feature you add when customers start asking what their traffic looks like. By then you have four tools that cannot see each other, and the numbers your business runs on live somewhere you do not control.
Built in week two, when there is nothing to measure, it stops being an analytics system. The invoice is computed from it. Errors are grouped by it. Eventually the platform will read it to find out whether what it wrote was any good. A customer getting a dashboard with nothing to install is a side effect.
Building something like this
I'm Jesse. I build platforms end to end, and I'm open to work. If this is the kind of engineering you need, get in touch.