EpiAttend: A transformer model of gene regulation combining single cell epigenomes with DNA sequence

Understanding cell type specific gene expression regulation requires models that integrate information across long genomic distances, such as enhancer-gene interactions spanning many tens of kilobases. Neural network models using deep convolutions and self-attention have achieved highly accurate prediction of cell type specific gene expression and other functional genomics measurements based on DNA sequence in local windows\citep{avsec,basenji2}. By contrast, leading models for linking enhancers with target genes take advantage of cell type specific epigenomes\citep{abc}. Here, we propose a framework for combining DNA sequence with epigenetic data from single cell sequencing within a neural network to predict cell type specific functional readouts such as mRNA expression. This approach has the potential to identify long-range gene-regulatory interactions, linking enhancers with genes based on both the epigenome and DNA sequence binding motifs.