Challenges of Binary Tokenization for Protein Generation

Protein structure tokenizers make 3D geometry accessible to sequence models and enable multimodal models over structure, sequence, and function, but at the cost of structural fidelity compared to continuous methods. At the same time, recent advancements in discrete visual autoregressive models have unlocked massive vocabulary scaling by adopting binary tokenization, closing this gap. In this work, we seek to know whether this method transfers to protein structure tokenization. After evaluations in reconstruction and generation across a variety of models and settings, we find that binary tokenization does not transfer cleanly to the protein structure setting and characterize where the failure modes arise.