Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

Existing depth-compression methods for large language models (LLMs) operate on layers individually or restrict grouping to adjacent or contiguous positions, implicitly assuming functional redundancy is local. We challenge this assumption by computing full pairwise Centered Kernel Alignment (CKA) similarity matrices across thirteen LLMs spanning six architecture families and 1.6B--14B parameters, revealing that non-adjacent layers exhibit 3.6--15.9$\times$ more high-similarity pairs than adjacent ones. We propose Fusion via Community Mining (FCM), which builds the full $L \times L$ similarity graph, applies spectral community detection to recover clusters of functionally equivalent layers, retains the highest-Block-Influence layer per cluster, and applies LoRA recovery. At 2$\times$ compression, FCM lowers mean perplexity below ShortGPT on twelve of thirteen models, with statistically significant improvements on ten under a Welch's $t$-test with Holm--Bonferroni correction (up to 50\% on Qwen3-4B, $p_\text{corr}<0.001$); two rows are ties and Mistral-7B is the lone loss ($+$14.3\%). On three anchor models, FCM also outperforms three recent adjacent-pair methods (FlattenGPT, SWM, DeltaLLM) by 7.8--35.3\% under matched calibration and recovery. The perplexity advantage transfers downstream: on a five-benchmark zero-shot reasoning suite (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA) over four anchor models, FCM matches or exceeds ShortGPT on 15 of 20 (model, benchmark) cells, and the 2$\times$-compressed Qwen2.5-7B realises 46\% fewer parameters, 42\% lower peak VRAM, and 84\% higher decode throughput on a single A100. Ablations confirm spectral grouping outperforms Fisher contiguous segmentation by 41\% and uniform partitioning by 51\%, and a compression sweep reveals a 2$\times$ crossover beyond which FCM's advantage grows monotonically and persists at the 14B scale. Code: \url{https://anonymous.4open.science/r/fcm-layer-fusion-D820}.