K-Blade Concept Structure vs. the Live Site's Self-Organizing Map: An Efficiency Comparison

Authors: Pat Parslow (Draft)

Third in this series, following "Algorithm, Not Metric" (Tests A–E) and "Overlapping Membership and Geometric-Algebra Borders" (Test F). This paper is a single, focused measurement: how does the full k-blade similarity/clustering/border pipeline compare, in wall-clock cost, to the (then-)live site's SOM-based concept atlas, on the identical corpus?

Abstract

TheThe parslow.net site's then-live "Cartographic Topological Map" was generated by a 40×40 Pure-Python Self-Organizing Map [1] trained for 30 epochs over broken-stick TF-IDF vectors [2]. We instrumented that exact pipeline (cache bypassed, so the measurement reflects a genuine cold build) and the full k-blade similarity-plus-clustering-plus-border pipeline from the two predecessor papers, on the same 160-page corpus, and timed both stage-by-stage. The SOM pipeline totals 802.1 seconds (13 minutes 22 seconds), of which SOM training alone accounts for 797.1 seconds (99.4% of the total). The k-blade pipeline totals 230 milliseconds. The SOM pipeline is 3,485× slower on this corpus. We explain the mechanism behind this gap — a fixed grid resolution vs. a corpus-size-scaling computation — and discuss what the SOM buys for that cost that the k-blade pipeline, as it stood at the time of this measurement, did not yet replace.

1. Method Applied / MethodologicalKnowledge with a 5–10 year half-life — stable practice

BothBoth pipelines were run on the same 160-page corpus, with embedding/vectorisation inputs read from cache where both pipelines legitimately share that cost (nomic-embed-text vectors were already computed for prior papers in this series; TF-IDF vectorisation [2] is SOM-specific and timed fresh). The SOM run explicitly bypassed som_atlas.py's own content-hash cache (which would otherwise skip training entirely on an unchanged corpus) so the measurement reflects a genuine cold build, the situation that matters whenever the corpus actually changes. Both pipelines were run in the same Python process invocation, on the same machine, back to back, to avoid any environment-difference confound.

2. Results Ephemeral / ToolingKnowledge that evolves in months to a year — check for updates

SOM pipeline (40×40 grid, 30 epochs, 1203-dim broken-stick TF-IDF)
text extraction	1,060.9 ms
TF-IDF vectorisation	2,802.0 ms
SOM training (30 epochs)	797,084.5 ms
U-matrix computation	1,164.1 ms
TOTAL	802,111.4 ms (802.1 s)

K-blade pipeline (768-dim nomic-embed-text, embeddings pre-cached)
build k-blade similarity matrix (all 12,720 pairs)	154.8 ms
graph-communities clustering	45.4 ms
GA border metrics (all cluster pairs)	30.0 ms
TOTAL	230.2 ms

SOMSOM training alone (797.1 s) is 99.4% of the SOM pipeline's total cost, and is 3,462× the k-blade pipeline's ENTIRE total. Overall, the SOM pipeline is 3,484.6× slower than the full k-blade pipeline on this corpus.

3. Why the Gap Is This Large FoundationalKnowledge that endures for decades — core principles

TheThe mechanism is structural, not an implementation inefficiency in either pipeline. The SOM's dominant cost is fixed grid resolution × epochs: a 40×40 grid is 1,600 nodes, each of which is updated (in varying strength, per the neighbourhood function — the standard competitive-learning update rule [1]) on every one of the 160 training vectors, for 30 full epochs, in pure Python with no vectorised bulk-array backend for the per-node weight update loop. Each epoch visits every training vector and updates all 1,600 nodes for each one, so the cost grows linearly with corpus size (and with vocabulary size, since the node weights have one entry per term) on top of the grid-size factor; for a fixed grid and epoch count it is a large constant multiplied by \(n\).what a SOM does each epoch: see the legacy page

The k-blade pipeline's dominant cost has a different shape: an \(O(n^2)\) pairwise similarity computation (12,720 pairs for 160 pages), one small SVD per pair, each \(O(r^2 d)\) rather than anything exponential in embedding dimension — the property established in the predecessor papers' Tests A–C. For two single-page blades \(r=1\), and the generalized sine reduces to \(\sin\theta\), the sine of the angle between the two embeddings. At page level the similarity is therefore a monotone function of cosine similarity (we confirm this numerically in Section 6); the higher-rank blades enter at cluster level, in the border metrics. The pairwise cost scales with corpus size rather than an arbitrarily chosen grid resolution, and at \(n=160\) it sits nowhere near the point where \(O(n^2)\) looks expensive. Both cost models are predictable from first principles. The 3,485× gap measured here reflects the scale of this corpus against this grid, and a substantially larger corpus or a much larger SOM grid would shift it. Section 6 re-measures at 357 pages.

4. What the SOM's Cost Bought, That the K-Blade Pipeline Did Not Yet Replace (At Time of Writing) Ephemeral / ToolingKnowledge that evolves in months to a year — check for updates

ThisThis was not a like-for-like comparison of equivalent outputs, and reporting the speed gap without this section would have overstated the finding. The SOM pipeline additionally produced: a full topological elevation surface (the U-Matrix) usable for pathfinding "terrain difficulty" between any two arbitrary points on the map (not just between existing pages); a fixed, stable 2D grid coordinate system that composed cleanly with the (then-)live site's tile/route rendering; and years of prior tuning (broken-stick feature selection, hazard-keyword elevation biasing, road/bridge-term generation) specific to this site's navigation UI. The k-blade pipeline, as built across this paper series, produced a similarity structure, a partition (optionally overlapping), and inter-cluster border metrics — a different, not strictly smaller, set of outputs. The natural continuation of this work (a k-blade-based 2D map projection, developed as a companion to this paper) was an attempt to recover the SOM's core navigational value from the k-blade structure directly, at a small fraction of the cost. That continuation subsequently became the live site's Concept Atlas, replacing the SOM entirely; see the production writeup for how each of these SOM-specific outputs (terrain, coordinates, road/bridge text) was carried over.

5. Limitations Ephemeral / ToolingKnowledge that evolves in months to a year — check for updates

• Only one corpus size (160 pages) and one SOM configuration (40×40, 30 epochs) were tested; the crossover point (if any exists at a realistic scale) between the two cost curves was not determined.
• The SOM implementation timed here was the site's actual Pure-Python implementation, not an optimised (e.g. NumPy-vectorised or GPU) SOM; a vectorised SOM implementation would likely close much of this gap, and the comparison should be read as "this specific pipeline as it was then deployed" rather than "SOMs are inherently this expensive."
• The SOM's cache (bypassed here deliberately) meant this 802-second cost was, in the site's actual operation at the time, paid only when the corpus content changed, not on every page load or every build — the practical cost-per-build in steady state was much lower than this cold-build figure and was not separately characterised.

6. Update: the Atlas at 357 Pages Ephemeral / ToolingKnowledge that evolves in months to a year — check for updates

TheThe live atlas now covers 357 pages, up from 160. We re-timed the production pipeline at that size, on the cached embeddings, without rerunning the SOM. Subsets of 40, 80, 160 and 320 pages, drawn at random from the same corpus, give the scaling curve; each figure is the best of three runs on the same machine for the similarity, graph and clustering stages, and the best of two single runs for the two layout stages.

Pages	Pairs	Similarity (ms)	Graph (ms)	Modularity (ms)	MDS (ms)	Force layout (ms)
40	780	10.6	0.4	5.1	35.0	10.9
80	3,160	39.1	1.2	13.7	74.7	24.1
160	12,720	155.5	2.1	33.3	292.0	81.5
320	51,040	646.2	6.3	113.1	1,012.2	457.4
357	63,546	812.7	7.8	137.5	1,133.1	537.7

Doubling from 160 to 320 pages multiplies the similarity stage by 4.2, MDS by 3.5 and the force layout by 5.6, which is the quadratic growth the pairwise structure predicts. The 160-page row reproduces the original measurement: similarity plus clustering comes to 191 ms here against 200 ms in Section 2 (the border stage, 30 ms originally, was not re-timed).

The stages comparable with Section 2 — similarity, graph construction and clustering — total 0.96 s at 357 pages. The full production build adds the MDS seed layout and the force-directed layout, for 2.63 s. The SOM training time scales linearly in corpus size, so extrapolating the measured 797.1 s from 160 to 357 pages gives roughly 1,780 s (our estimate, not a measurement). That puts the SOM at about 1,850× the comparable k-blade stages and about 680× the full production build. The gap has narrowed from 3,485× because the k-blade cost grows quadratically while the SOM's grows linearly.

The similarity stage is a Python loop of per-pair SVDs. Because the single-page generalized sine is a function of cosine, as the next paragraph shows, the whole matrix can be produced by a single matrix product, and the 813 ms is an implementation cost rather than a property of the method.

Across all 63,546 pairs, the largest difference between the k-blade similarity and \(1-\sqrt{1-\cos^2\theta}\) is \(8.9\times10^{-16}\), and the Spearman correlation with \(|\cos\theta|\) is 1.000000. Page-level k-blade similarity carries the same ranking information as cosine similarity.

7. Conclusion

OnOn this 160-page corpus, the full k-blade similarity-clustering-border pipeline was 3,485× faster than a cold build of the site's then-existing 40×40 SOM pipeline, because the two approaches have fundamentally different cost models (fixed-grid×epochs vs. corpus-size-scaling pairwise linear algebra) rather than because one is a more efficient implementation of the same computation. This margin made a k-blade-based concept map practical to rebuild far more frequently than the SOM had been, which is directly relevant to the companion visualisation work (Test G) that turned this structure into a navigable map artefact — and, subsequently, into the site's live production atlas.cf. the atlas as it runs today

Appendix: Reproducibility

Code:Code: tools/site/concept_experiments/kblade_vs_som_timing.py. Runs the (then-)live tools/site/som_atlas.py's own build_tfidf_vectors and PurePythonSOM class directly (cache bypassed), and the k-blade pipeline from ga_kblade_hierarchy.py / ga_kblade_graph_communities.py, on the same cached 160-page embedding data (ensemble_cache.npz) used throughout this paper series. The figures in Sections 1–5 are read directly from that script's stdout. The Section 6 figures come from tools/site/concept_experiments/kblade_atlas_rescale.py, which loads the 357 pages of the shipped atlas with their cached embeddings (ensemble_cache.npz plus ensemble_cache_extra.npz); the one SOM figure there, 1,780 s, is the linear extrapolation described in that section.

Related Topics

• Algorithm, Not Metric — the predecessor paper establishing the k-blade clustering pipeline timed here, including why it needed a non-greedy algorithm to work at all.
• Overlapping Membership and Geometric-Algebra Borders for a Concept Hierarchy — Test F, run on the same corpus, extending the timed pipeline to multi-membership clusters and GA-native border metrics.
• K-Blades vs. Self-Organizing Maps: How the Live Concept Atlas Works Now — the production writeup of what this timing comparison led to: the SOM engine measured here has since been fully retired.

References

1. T. Kohonen, Self-Organizing Maps, 3rd Edition. Springer-Verlag, Berlin, Heidelberg, 2001. https://link.springer.com/book/10.1007/978-3-642-97966-8
2. G. Salton and C. Buckley, "Term-Weighting Approaches in Automatic Text Retrieval," Information Processing & Management, vol. 24, no. 5, 1988, pp. 513–523. https://doi.org/10.1016/0306-4573(88)90021-0