Model Evolution Helps to Decipher the Process of Transcriptional Regulation in Human Cells

March 10, 2025 By Stuart P. Atkinson

Multi-panel figure describing the data and training used for the model, and headline results — Motif masking and fill-in strategies coupled with an F/p model integrate chromatin accessibility and expression to infer TF–TF interactions and build a structural interaction catalog. Scatterplots show predicted versus observed gene expression, cell-type prediction accuracy, and correlations with TSS accessibility and gene activity (R² and r values). From Fu, Mo, and Buendia et al.

Model Evolution Helps to Decipher the Process of Transcriptional Regulation in Human Cells

While transcriptional regulation underpins the diversity of biological and pathological processes, our understanding of this process remains surprisingly incomplete. Cell-specific transcriptional profiles arise from myriad protein-protein and protein-DNA interactions taking place in a background of differing epigenetic conditions. The reported clustering of transcription factor (TF) binding motifs (Vierstra et al.) highlights the homology of DNA-binding domains and the low combinatorial variability regarding regulatory interactions; however, our understanding of transcription regulation remains limited to specific cell types. Furthermore, we still do not know if combinatorial interactions of TFs determine cell-specific gene expression profiles. Overall, we appear to understand little regarding the critical process of transcriptional regulation.

Developing fine-tuned prediction methods based on sequence data and trained on specific human cell types (Zhou et al., Kelly 2020, and Zhang et al.) represented a first step towards an improved understanding of transcriptional regulation; recently, the implication of foundation models machine/deep learning models trained on vast datasets for application to range of use cases have fostered generalizability and improved utility (OpenAI. GPT-4 technical report and Lin et al.). More recent developments in single-cell Theodoris et al., Cui et al., and Hao et al.) have reported the encoding of transcriptomic profiles within single models to enable downstream tasks. Will the report of a foundation model describing how transcription emerges from the chromatin landscape represent the next evolutionary step?

Understanding relationships between chromatin and transcriptional output remains tricky, given that approaches generally require separate epigenetic and transcriptomic assays in distinct cell samples and reintegrating these diverse datasets. Parallel analysis of individual cells for RNA expression and DNA from targeted tagmentation by sequencing or " Paired-Tag Epigenome Technologies generates joint epigenetic and gene expression profiles at the single-cell resolution and detects histone modifications and RNA transcripts in individual nuclei with efficiencies similar to single-nucleus RNA-seq/ChIP-seq assays. Paired-Tag technology can enable researchers to advance our understanding of transcriptional regulation and improve disease management.

Researchers from the laboratories of Xi Fu, Raul Rabadan (Columbia University), and Eric P. Xing (MBZUAI/Carnegie Mellon University) knew that current computational models of transcriptional regulation lacked the generalizability required to support extrapolation to "unseen" cell types and conditions, which would aid the appreciation of gene regulation at a deeper level and perhaps extend to identifying new targets in a variety of diseases/disorders. In their recent Nature study, the authors introduce GET - general expression transformer Fu, Mo, and Buendia et al.). In part one of this series, Epigenome Technologies offers a brief overview of the development of GET before describing the practical applications of this exciting technological advance.

Cartoon of masking and training protocol — Schematic of GET’s input matrix: each gene is represented by fixed TF-motif binding scores across 200 gene-proximal ATAC peaks (aggregated into 282 clusters, grey) combined with cell-type-specific pseudobulk ATAC signal (orange) to predict gene expression.

Developing and Benchmarking the General Expression Transformer

GET focuses on characterizing a local genomic region containing regulatory elements by understanding cell-specific TF binding and chromatin accessibility, which provides a proxy for PolII-driven gene expression within each element (approximated from RNA-seq data)
The method of encoding each gene as a structured matrix represents a distinctive innovation; GET defines a gene by the local regulatory landscape instead of assigning each gene a unique identifier

Specifically, a matrix with dimensions of 200 rows 283 columns represents each gene
The rows represent a series of contiguous windows or "bins" spanning a genomic region around the transcription start site; each row represents a localized region of chromatin accessibility
The columns aggregate TF motif binding scores into 282 clusters with an additional channel that captures underlying chromatin accessibility

An individual matrix entry quantifies the likelihood of a TF binding event (or the accessibility level) at that precise bin

A high value at a particular row and column may indicate robust predicted binding for a specific TF motif within a given region of chromatin accessibility, while lower scores suggest weaker binding/less accessibility
This detailed representation enables GET to "define" a gene by a spatially resolved profile of regulatory features a property leveraged for predicting reporter assays

A regulatory element-specific architecture supports a self-supervised pre-training step that allows GET to learn how regions and features interact across cell types

Masking out random regulatory elements trains GET to predict TF binding/chromatin accessibility and subsequent PolII activity/gene expression
A portion (typically up to 50%) of the matrix entries are masked out using a learnable token, with GET aiming to impute - or accurately predict - missing values based on the context provided by surrounding entries
Imputing missing data forces the model to learn local correlations and dependencies within the matrix
For instance, the model can capture relationships when specific TF motifs co-occur in neighboring bins or when specific accessibility patterns predict binding events in adjacent regions
The design of GET supports the use of chromatin accessibility data without paired gene expression data to improve the diversity of training data regulation information

GET pre-training employs single-cell assay for transposase-accessible chromatin with sequencing (scATAC-seq) data from 213 human cell types from the fetus and adult (Zhang et al., Joung et al., and Domcke et al.)

Fine-tuning utilizes gene expression data for 153 cell types from multiomics or separate single-cell RNA-seq analyses (Cao et al. and The Tabula Sapiens Consortium)

An assessment of GET's ability to accurately predict gene expression in unseen cell types reveals a high correlation between GET's predicted and the observed expression values for a "left-out" cell type (in this case, astrocytes)

GET significantly outperforms related technologies, including more straightforward machine learning approaches trained in the same manner on the same data

The authors note the elevated level of generalizability, as GET functions on adult cell types when trained solely on fetal data and employing distinct sequencing platforms and other experimental assays
An examination of zero-shot prediction capacity (recognizing/categorizing without observing examples beforehand) of gene expression-driving regulatory elements in unseen cell types reveals GET's ability to accurately predict the outcomes of a lentivirus-based massively parallel reporter assay in lymphoblast cells

In this case, GET pre-training employs chromatin accessibility and gene expression data from lymphoblast cells but did not expose GET to reporter assay data

Benchmarking against the deep learning architecture known as "Enformer" (Avsec et al.) in lymphoblast cells reveals that GET made more accurate predictions and scaled better, with predicted regulatory elements displaying meaningful enrichment of histone modifications and TF binding sites

Training Enformer employed TF binding, histone modification, gene expression, and chromatin accessibility data and functioned better in certain instances (enhancers and repressed regions); however, GET displays significant advantages in terms of computational cost

Explication of the GET transformer architecture — Transformer architecture for GET: a 200×283 input is linearly expanded, processed through 12 self‐attention layers (12 heads, dₖ=dᵥ=64) with GeLU‐activated feed‐forward blocks, then linearly collapsed with softplus to produce a 200×1 output.

Can Chromatin Accessibility Alone Suffice When Predicting Gene Expression?

While GET leverages chromatin accessibility as a primary proxy for active regulatory regions, this approach prompts a pivotal question: can chromatin accessibility data explain gene expression levels, or do we require additional epigenetic information, such as histone modification profiles? Said modifications function directly by recruiting transcriptional activators or indirectly through chromatin remodeling complexes, suggesting that accessibility lays the groundwork for gene activation, but the full orchestration of transcription depends on a complex interplay of structural/chemical signals.

Multi-panel figure showing results of expression prediction versus a lentiviral assay — Scatterplots comparing predicted versus observed lentiMPRA log₂(RNA/DNA) for promoters, enhancers, repressive/quiet regions, and all sequences: GET-MPRA (top row) versus Enformer-MPRA (bottom row), with Pearson’s r, slope, and n indicated. From Fu, Mo, and Buendia et al.

General Expression Transformer The Highlights

Overall, the authors report how GET provides experimental-level accuracy in gene expression prediction in seen and unseen cell types employing chromatin accessibility data and sequence information, displays adaptability across sequencing platforms and assays, offers zero-shot prediction of reporter assay readouts, and outperforms previous state-of-the-art models when identifying cis-regulatory elements.

Future studies may be supported by robust single-cell datasets provided by Paired-tag an analytical platform that creates joint epigenetic and gene expression profiles at single-cell resolution and detects histone modifications and RNA transcripts in individual nuclei. The Bing Ren lab developed Paired-tag, and Epigenome Technologies offers optimized Paired-tag kits and services to epigenetics researchers an exclusive license from the Ludwig Institute for Cancer Research.

See Nature, January 2025 for more on the development of GET as an interpretable foundation model for transcriptional regulation, and stay tuned to Twitter, Bluesky, and LinkedIn to keep up to date with all the new epigenetics studies and the Epigenome Technologies website for part 2. In the meantime, check out our Products and Services pages to see how Epigenome Technologies can elevate your research today.