Leaderboard — VUS-PR

Lite profile — preview

These numbers come from the lite subset (CI-sized: a handful of entities per dataset), not the full benchmark matrix. Means are shown with entity-bootstrap 95% CIs and an entities count — a mean over few entities with a wide CI is weak evidence. Do not quote ranks from this page without those qualifiers.

Primary metric: VUS-PR (Paparrizos et al., VLDB 2022). Values are seed averages; cell shading follows the metric value, row tint and badge follow the generation. PA-F1 is intentionally not shown (CLAUDE.md §4).

Friedman test across the 21 models with complete coverage of the 13 shared entities: p = 1.71e-09 — model ranking differences are statistically significant overall; see the critical-difference diagram for which pairs are separable.

Model summary (sorted by mean rank)

generation	model	mean vus_pr [95% CI]	mean rank	entities	runtime (s)	config
Gen 4	mtad_gat	0.821 [0.72, 0.89]	5.00	3	5.85	10ce709c421b
Gen 3	lstm_p	0.812 [0.71, 0.91]	5.33	3	2.38	2d11f154dbe4
Gen 2	sub_knn	0.455 [0.32, 0.59]	7.15	13	9.24	1622386787b5
Gen 1	sub_pca	0.439 [0.31, 0.57]	7.46	13	13.26	3a8820e97431
Gen 3	lstm_ad	0.782 [0.63, 0.89]	7.67	3	2.66	0dfacdce9d7b
Gen 4	tranad	0.763 [0.66, 0.91]	8.33	3	20.63	2e0a261787de
Gen 3	usad	0.746 [0.72, 0.79]	9.33	3	4.58	0a248a10c2d7
Gen 3	ae	0.734 [0.72, 0.75]	9.67	3	1.90	00565c2c3da2
Gen 2	ocsvm	0.404 [0.26, 0.56]	10.08	13	2.16	19dda5b3ab8b
Gen 0	ensemble_simple	0.383 [0.25, 0.52]	10.23	13	19.83	03504b13be43
Gen 1	sesd	0.406 [0.28, 0.55]	10.38	13	157.21	0b4ff0f6ef56
Gen 1	pca_t2spe	0.428 [0.27, 0.59]	10.54	13	0.10	05d8070873bb
Gen 5	mamba_tsad_fixed	0.500 [0.37, 0.62]	11.00	7	19.94	0465248a5e25
Gen 1	hotelling_t2	0.437 [0.28, 0.59]	11.15	13	0.04	2803a14210c8
Gen 2	lof	0.411 [0.26, 0.56]	11.46	13	3.19	0d91f8dc49f0
Gen 1	ewma	0.413 [0.26, 0.57]	11.54	13	0.20	0454ea03f379
Gen 2	knn	0.427 [0.27, 0.58]	11.62	13	0.54	30938e6805a4
Gen 1	poly	0.378 [0.23, 0.55]	11.85	13	0.02	04f78382c778
Gen 1	stl_residual	0.403 [0.26, 0.55]	12.08	13	101.17	13f77ab717d1
Gen 1	zscore	0.380 [0.23, 0.54]	13.54	13	0.03	247f60d04790
Gen 4	dcdetector	0.647 [0.43, 0.80]	14.33	3	2.11	02770db3600f
Gen 4	timesnet	0.645 [0.54, 0.81]	15.00	3	2.41	047d78f5691c
Gen 2	iforest	0.332 [0.22, 0.46]	15.38	13	0.64	053d1ce8d6cd
Gen 2	hbos	0.296 [0.19, 0.42]	16.46	13	0.45	0ac2775a90c3
Gen 3	vae_donut	0.575 [0.40, 0.76]	17.00	3	2.69	4b166e2b2d78
Gen 5	mamba_tsad_faithful	0.318 [0.20, 0.43]	17.29	7	21.11	103a30200cbe
Gen 3	omni_anomaly	0.609 [0.56, 0.64]	17.33	3	7.48	0bbd75197861
Gen 2	matrix_profile	0.287 [0.18, 0.41]	18.31	13	30.60	169bf64ba146
Gen 4	anomaly_transformer	0.613 [0.45, 0.78]	18.67	3	4.28	0302c3c4f1ba
Gen 2	spectral_residual	0.200 [0.11, 0.33]	18.85	13	0.17	1a3ce6927c20
Gen 1	cusum	0.232 [0.14, 0.34]	19.08	13	0.24	02df36fe32e4
Gen 1	online_ewma	0.224 [0.12, 0.36]	19.15	13	0.79	1d197cc21867
Gen 1	online_cusum	0.208 [0.12, 0.32]	19.69	13	1.16	02b785172714
Gen 0	dummy	0.179 [0.10, 0.29]	21.69	13	0.00	0051646a3b58
Gen 4	gdn	0.420 [0.21, 0.62]	26.33	3	6.84	0b1bd833ee60
Gen 3	dagmm	0.291 [0.16, 0.52]	29.33	3	2.37	12266f472566

Model × dataset

generation	model	mba	mgab/1	mgab/2	nab/ realAWSCloudwatch_ec2_cpu_utilization_24ae8d.csv	nab/ realKnownCause_machine_temperature_system_failure.csv	psm	skab/valve1_0	smd/ machine-1-1	smd/ machine-1-6	smd/ machine-2-1	smd/ machine-3-2	smd/ machine-3-7	synthetic
Gen 4	mtad_gat	–	–	–	–	–	–	0.893	0.717	–	–	–	–	0.852
Gen 3	lstm_p	–	–	–	–	–	–	0.909	0.709	–	–	–	–	0.817
Gen 2	sub_knn	0.690	0.091	0.105	0.307	0.516	0.495	0.908	0.744	0.587	0.501	0.114	0.262	0.592
Gen 1	sub_pca	0.731	0.067	0.065	0.272	0.196	0.531	0.721	0.762	0.644	0.416	0.274	0.386	0.648
Gen 3	lstm_ad	–	–	–	–	–	–	0.889	0.628	–	–	–	–	0.830
Gen 4	tranad	–	–	–	–	–	–	0.913	0.661	–	–	–	–	0.715
Gen 3	usad	–	–	–	–	–	–	0.793	0.724	–	–	–	–	0.719
Gen 3	ae	–	–	–	–	–	–	0.731	0.725	–	–	–	–	0.747
Gen 2	ocsvm	0.809	0.059	0.064	0.222	0.381	0.495	0.387	0.673	0.790	0.436	0.108	0.042	0.782
Gen 0	ensemble_simple	0.767	0.063	0.073	0.201	0.305	0.569	0.732	0.510	0.611	0.348	0.114	0.101	0.587
Gen 1	sesd	0.889	0.057	0.066	0.206	0.182	0.378	0.435	0.747	0.296	0.357	0.310	0.681	0.674
Gen 1	pca_t2spe	0.774	0.062	0.064	0.143	0.637	0.527	0.831	0.581	0.816	0.246	0.069	0.123	0.697
Gen 5	mamba_tsad_fixed	–	–	–	–	–	–	0.378	0.748	0.663	0.429	0.192	0.477	0.611
Gen 1	hotelling_t2	0.770	0.062	0.064	0.143	0.637	0.544	0.833	0.509	0.812	0.255	0.060	0.301	0.688
Gen 2	lof	0.696	0.057	0.065	0.198	0.461	0.473	0.900	0.602	0.545	0.453	0.062	0.055	0.782
Gen 1	ewma	0.716	0.062	0.064	0.169	0.637	0.420	0.933	0.517	0.569	0.509	0.103	0.036	0.637
Gen 2	knn	0.793	0.062	0.063	0.143	0.532	0.518	0.848	0.574	0.775	0.305	0.064	0.203	0.672
Gen 1	poly	0.908	0.064	0.063	0.169	0.136	0.429	0.403	0.598	0.682	0.197	0.103	0.250	0.919
Gen 1	stl_residual	0.888	0.057	0.066	0.177	0.182	0.444	0.548	0.648	0.671	0.203	0.061	0.623	0.675
Gen 1	zscore	0.765	0.062	0.064	0.143	0.637	0.431	0.674	0.466	0.678	0.217	0.064	0.038	0.706
Gen 4	dcdetector	–	–	–	–	–	–	0.430	0.713	–	–	–	–	0.798
Gen 4	timesnet	–	–	–	–	–	–	0.542	0.579	–	–	–	–	0.814
Gen 2	iforest	0.792	0.059	0.064	0.170	0.568	0.476	0.376	0.347	0.466	0.309	0.088	0.100	0.499
Gen 2	hbos	0.765	0.060	0.063	0.214	0.632	0.450	0.370	0.286	0.371	0.282	0.087	0.060	0.206
Gen 3	vae_donut	–	–	–	–	–	–	0.571	0.396	–	–	–	–	0.756
Gen 5	mamba_tsad_faithful	–	–	–	–	–	–	0.559	0.340	0.384	0.324	0.099	0.109	0.414
Gen 3	omni_anomaly	–	–	–	–	–	–	0.563	0.622	–	–	–	–	0.641
Gen 2	matrix_profile	0.635	0.450	0.522	0.139	0.065	0.313	0.493	0.096	0.199	0.063	0.062	0.113	0.579
Gen 4	anomaly_transformer	–	–	–	–	–	–	0.782	0.606	–	–	–	–	0.451
Gen 2	spectral_residual	0.803	0.066	0.064	0.142	0.095	0.312	0.418	0.114	0.222	0.075	0.076	0.026	0.186
Gen 1	cusum	0.677	0.059	0.064	0.218	0.110	0.310	0.521	0.167	0.333	0.136	0.044	0.030	0.343
Gen 1	online_ewma	0.855	0.062	0.063	0.116	0.097	0.313	0.440	0.117	0.212	0.073	0.068	0.022	0.476
Gen 1	online_cusum	0.679	0.059	0.064	0.197	0.184	0.288	0.453	0.111	0.139	0.051	0.100	0.021	0.357
Gen 0	dummy	0.685	0.062	0.065	0.137	0.095	0.302	0.364	0.105	0.212	0.072	0.063	0.022	0.140
Gen 4	gdn	–	–	–	–	–	–	0.433	0.206	–	–	–	–	0.620
Gen 3	dagmm	–	–	–	–	–	–	0.515	0.197	–	–	–	–	0.161

Reproduce with python benchmarks/run_all.py --profile configs/lite.yaml — each row's config hash resolves to its full configuration in the results JSON.