0

The Performance Toolkit already logs every iteration as a raw row in "BCPT Log Entry": duration in ms, number of SQL statements, operation, session, RunID. The reporting layer then reduces each suite line to an iteration count, a total, and an average. No other statistic is surfaced - not on the BCPT Suite Lines page, not in the API.


An average cannot describe a latency distribution. Two operations with the same mean can differ by an order of magnitude in their slow percentiles, and the slow percentiles are where users feel regressions first. A p99 regression that doubles the tail moves the mean by a few percent and stays invisible in the current report.


Real numbers from a 5-minute BCPT run on a 27.1 container (w1) (3 scenarios, 2 sessions each, 1,670 iterations):

  • Scenario A: 568 iterations. Average 43 ms. Distribution: min 11, median 44, p95 63, p99 169, max 213 ms.
  • Scenario B: 566 iterations. Average 12 ms. Distribution: min 7, median 11, p95 16, p99 38, max 112 ms.
  • Scenario C: 536 iterations. Average 47 ms. Distribution: min 30, median 43, p95 69, p99 121, max 191 ms.


Today's report for scenario A is "43 ms average". The distribution says 1 in 100 iterations costs 169-213 ms. The average is identical for a steady 43 ms operation and for this one.


Proposal: aggregate "BCPT Log Entry" per suite/version/line/operation into min, median, p95, p99, max duration, and surface it (1) as columns on BCPT Suite Lines next to Average Duration, reusing the existing base-version comparison pattern so p99 regressions show in A/B runs, and (2) in the results API so pipelines can gate on p95/p99 instead of the mean.


The toolkit already has everything needed: the raw rows exist, and "BCPT Log Entry" already carries a key ordered by suite, version, line, operation and duration. Reading in that order delivers each group's durations pre-sorted, so nearest-rank percentiles are a single ordered scan. No new storage, no schema change.


I have implemented this as a per-tenant extension against 27.1 and verified it: every figure above matches T-SQL PERCENTILE_DISC computed independently over the same rows. No extension can provide this cleanly for the ecosystem, per-tenant or AppSource alike: "BCPT Log Entry" is Access=Internal and the toolkit pages are Extensible=false, so anything outside the toolkit has to read the table by numeric RecordRef and rebuild the UI beside it. That is why this belongs upstream.

Category: Development
STATUS DETAILS
New