Prices

• Every value comes from the provider's public pricing page or docs. We store each value's source URL and the date we fetched it (raw JSON, CC BY 4.0).
• When a page doesn't state something (billing granularity, say, or isolation mechanism), we mark it unverified instead of guessing.
• We use list prices for the default region and the cheapest self-serve plan that allows the workload. Regional multipliers, taxes, committed-use discounts and enterprise pricing are excluded.

Cost per typical task

• Code-agent task: 2 vCPU / 4 GiB sandbox for 5 minutes at 10% CPU utilization (most time spent waiting on the LLM). Mirrors Vercel's own "AI code validation" example.
• Browser-agent task: One 3-minute cloud browser session, headless where the provider prices it separately, with no proxy, CAPTCHA or LLM costs.
• We apply each provider's billing rules as published: wall-clock or active CPU, minimum increments (for example Vercel's 1-minute memory minimum, or Browser Use's whole-minute rounding), Modal's 1 core = 2 vCPU, and Cloudflare's fixed instance types (we pick the smallest one that fits).
• Results are usage cost per run. Plan fees, included credits, storage, egress, proxies, CAPTCHA and LLM tokens are listed separately and never netted in.
• The billing rules we apply are stored per provider as costModel in the public data. The calculator runs the same cost model in your browser.

Change detection

• A weekly GitHub Action in latent-signal/coldtally-data fetches every pricing page, extracts the text and price tokens, and diffs them against the last snapshot. It also checks that every price we publish still appears on the page.
• Any difference opens a pull request with draft changelog entries marked needs-review. A human checks the source before anything is published to the changelog, the RSS feed or the newsletter.

Speed (secondary)

• ColdTally's core product is price and change tracking. The startup-time boards are a small secondary check, not a speed leaderboard.
• Schedule and sample. A GitHub Action in latent-signal/coldtally-data runs daily at 06:41 UTC: per provider, 1 warm-up run (discarded) and 10 measured runs, one at a time, with providers in random order. We publish the median and p95 of the successful runs, the success count, and each provider's SDK version. Manual runs can use a smaller sample; the board always prints the sample size it shows.
• What's timed. Sandboxes: from the create call until echo ok returns in the new sandbox (create + first command). Browsers: from the create-session call through a Playwright CDP connect until our fixed target page (coldtally.com/bench-target/) reaches DOMContentLoaded (create + connect + page load). Destroy and release times are recorded separately but not charted. Cloudflare Sandbox runs inside a Worker, so it is timed by our open benchmark Worker on Cloudflare, in ColdTally's own account; that excludes network latency from the runner, so it isn't directly comparable with the other sandboxes. We also publish its full round trip from the runner. Because the two types time different steps, sandbox and browser numbers aren't comparable with each other.
• Where it runs. On a GitHub-hosted Linux runner. Those runners aren't pinned to a region; we record the runner's approximate location (from an IP lookup) with each run. Provider regions are pinned only where the free plan allows it without a price multiplier, otherwise the provider picks placement and the board says "not pinned".
• Instance sizes differ. Each provider runs at its default size unless stated (for example, Modal is set to 1 core and 4 GiB). Startup time can depend on size and image, so read small differences with care.
• Cleanup. Every sandbox and session is destroyed in a finally block, and a sweep before and after each provider (by tag where the provider supports tags) removes any leftovers. Runs use free credits or ColdTally's own accounts.
• Providers without results aren't shown as rows; they're listed as "not yet measured". Raw results: data/bench/latest.json; harness: bench/.
• For broad speed coverage we credit and link to ComputeSDK's open benchmarks. Their data is MIT-licensed (github.com/computesdk/benchmarks). Anything we show from it is attributed to them and labelled with their run date. Note that ComputeSDK also sells private benchmarking to providers. We report their public data as-is.
• Providers whose terms forbid publishing benchmarks are listed with pricing only. Currently that's Blaxel.

Independence policy

1. Rankings aren't for sale. Order is always by computed cost (or by the metric shown). Sponsorship never changes inclusion, order, data or method.
2. No paid private tests. We don't sell private, early or "stealth" runs to the vendors we track, and we don't coordinate releases with them.
3. No vendor-paid content. Nothing in the changelog or newsletter is written or paid for by a tracked vendor.
4. Sponsor disclosure. Sponsored placements are always labelled. Vendors we rank can't sponsor the comparison tables or the changelog. All current sponsors are listed on the sponsor page.
5. Right of reply, not right of edit. Vendors can report errors with a source. Email corrections@coldtally.com with the provider, the page or field that is wrong, and a link to the source that shows the correct value. Or use the public correction form on GitHub. We fix verifiable mistakes and log the fix publicly. Missing a provider? See what gets one in.
6. Reproducible. The data, pricing scripts and benchmark harness are public in latent-signal/coldtally-data (code MIT, data CC BY 4.0). The site itself is not open source.
7. Funding disclosure. Any credits or sponsorship that pay for benchmark runs, from a provider or a community sponsor, are listed here with who, what and since when. Credits never affect rankings or numbers: covered providers run the same open harness and their results publish like everyone else's. So far: none. Daily runs use providers' standard free credits, credits ColdTally bought itself (a $5 Browser Use top-up), or ColdTally's own accounts. Providers we'd measure with help are listed under Help us measure.

Who runs ColdTally

ColdTally is made by Latent Signal Labs, LLC. The independence policy above applies to everything we publish.

Disclosure. Mark Phelps, who runs ColdTally, works at Cloudflare. Cloudflare Sandbox and Cloudflare Browser Run are tracked on the same terms as every other provider: rates come from Cloudflare's public pricing pages, startup times come from the same open harness and daily run, and Cloudflare has no say in rankings or data beyond the right of reply every vendor has. One difference is shown on the boards: both are timed inside ColdTally's benchmark Worker, so their startup times leave out network latency, appear in grey and take no rank.