What a p-value actually tells you

You

I keep hearing p-values are misused. What do they mean?

ChatGPT

The probability of data at least this extreme if the null hypothesis were true. That is all. It is not the probability the hypothesis is false, and a small p with a tiny effect size usually means a large sample rather than an important finding.

Tool steps: 3 (reasoning summary, call to python, execution output)

reasoning summary

Checked the query plan, then compared index options.

call to python

rows = run_sql("EXPLAIN (ANALYZE) SELECT ...")
print(rows[:3])

execution output

Seq Scan on events  (cost=0.00..812431.00 rows=41203122 width=16)
  Rows Removed by Filter: 38996878
Planning Time: 0.214 ms
Execution Time: 11842.663 ms

You

How long does this usually take in practice?

ChatGPT

An afternoon to do it properly the first time, then minutes when it recurs. Most of the first afternoon goes into discovering which of the defaults do not match your situation.