Communication efficiency is a major challenge in federated learning. In client-server schemes, the server constitutes a bottleneck, and while decentralized setups spread communications, they do not reduce them. We propose a communication efficient semi-decentralized federated learning algorithm for feature-distributed data. Our multi-token method can be seen as a parallel Markov chain (block) coordinate descent algorithm. In this work, we formalize the multi-token semi-decentralized scheme, which subsumes the client-server and decentralized setups, and design a feature-distributed learning algorithm for this setup. Numerical results show the improved communication efficiency of our algorithm.