- Broad range of tissues
- Excluded cell types with high mutational burden
- Single-cell data may be more precise for inference of gene regulatory networks, since no averaging over cells
- Expression values are rank-encoded, normalized to expression across the corpus of cells
- De-prioritizes ubiquitously expressed housekeeping genes
- Genes that are highly specific to certain cell types will be given more weight
- Input to model is full set of rank-encoding for all genes
- No sequence to expression model
- Architecture: six transformer encoder units
- Self-supervised masked learning objective
- 15% of genes masked during pre-training
- 256-dimensional embedding for each gene
- Context-specific → depends on other expressed genes
- Evaluated it by in silico re-programming fibroblasts by upregulating Yamanaka factors (OCT4, SOX2, KLF4 and MYC)
- Integrate gene embeddings to create cell embeddings
-
Cell type annotation: claim performance is boosted in a variety of tissues (supplementary data)
-
Dosage-sensitivity:
-
Fine-tuned on 10,000 random single-cell transcriptomes to distinguish dosage-sensitive vs dosage insensitive transcription factors (previously published gene sets)
-
Large boost in performance compared to SVMs and random forests
- (unclear what data was used for this plot – does it overlap the fine-tuning training set?)
-
Further evaluations on newly reported neurodevelopmental dosage-sensitive genes
-
In silico deletion evaluation in fetal cardiomyocytes
- Genes whose deletion predicted to have most deleterious effect enriched for human phenotypes
- Not the most convincing evaluation, hard to know what to make of it
- Genes whose deletion predicted to have most deleterious effect enriched for human phenotypes
-
-
Chromatin dynamics predictions:
- Fine-tuned on transcriptomes from 15,000 embryonic stem cells to identify bivalent chromatin
-
Bivalent chromatin = marked by both repressing (H3K4me3) and activating (H3K27me3) regulators
-
Clear boost in performance compared to SVMs and random forest
-
Performance generalized to unseen regions of the genome, and when only training on 56 loci
-
- Fine-tuned on transcriptomes from 15,000 embryonic stem cells to identify bivalent chromatin
-
Network dynamics prediction
- References interesting previous work mapping gene regulatory network for NOTCH (Theodoris et al. 2021)
- Fine-tuned Geneformer to distinguish central vs peripheral factors on single-cell data from the Heart Atlas
- Significant boost in performance
- Investigated threshold for minimum data needed for fine-tuning, got down to 884 samples from healthy vs dilated aortas
- Looked at attention weights to try and draw conclusions about what is learned in each layer
-
In silico gene network analysis
- Deleted one gene, looked at which others were most affected
-
In silico treatment analysis
- Fine tuned Geneformer to distinguish cardiomyocytes from non-failing hearts and hearts affected by hypertrophic or dilated cardiomyopathy
- Determined genes whose in silico deletion or activation in non-failing hearts significantly shifted the cell embedding towards disease state
- In silico treatment analysis in fine-tuned disease models to evaluate activation and inhibition of specific pathways
- Experimental validation: CRISPR-knockout of Geneformer-predicted targets GSN and PLN in TTN het cells ⇒ significantly improved contractile stress