Recent advances in implicit neural representation (INR)-based video coding havedemonstrated its potential to compete with both conventional and other learning-based approaches. With INR methods, a neural network is trained to overfit avideo sequence, with its parameters compressed to obtain a compact representationof the video content. However, although promising results have been achieved,the best INR-based methods are still out-performed by the latest standard codecs,such as VVC VTM, partially due to the simple model compression techniquesemployed. In this paper, rather than focusing on representation architectures, whichis a common focus in many existing works, we propose a novel INR-based videocompression framework, Neural Video Representation Compression (NVRC),targeting compression of the representation. Based on its novel quantization andentropy coding approaches, NVRC is the first framework capable of optimizing anINR-based video representation in a fully end-to-end manner for the rate-distortiontrade-off. To further minimize the additional bitrate overhead introduced by theentropy models, NVRC also compresses all the network, quantization and entropymodel parameters hierarchically. Our experiments show that NVRC outperformsmany conventional and learning-based benchmark codecs, with a 23% averagecoding gain over VVC VTM (Random Access) on the UVG dataset, measuredin PSNR. As far as we are aware, this is the first time an INR-based video codecachieving such performance.