Last updated: 2023-08-30

Checks: 2 0

Knit directory: lab-notes/

This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Repository version: ba8cc1d

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version ba8cc1d. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated: 


Ignored files:
    Ignored:    analysis/doc_prefix.html

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/midway3.Rmd) and HTML (docs/midway3.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File Version Author Date Message
html d9c3393 1onic 2023-08-28 Build site.
html d4afa3b 1onic 2023-07-12 Build site.
html 289324b 1onic 2023-07-12 Build site.
html be52451 1onic 2023-07-12 Build site.
html a73ac21 1onic 2023-06-13 Build site.
html 41874f0 1onic 2023-06-13 Build site.
html 75989aa 1onic 2023-06-13 Build site.
html 14b2f73 1onic 2023-06-13 Build site.
html 6f3f6a4 1onic 2023-05-17 Build site.
html 5192404 1onic 2023-05-17 Build site.
html 93a8936 1onic 2023-05-10 Build site.
html 454b232 1onic 2023-05-10 Build site.
html 6a1bef0 1onic 2023-05-10 Build site.
html b8fbbd1 1onic 2023-05-10 Build site.
html e5648db 1onic 2023-05-10 Build site.
html c0c4bf0 1onic 2023-04-26 Build site.
html 9fbd5e6 1onic 2023-04-26 Build site.
html 1478db1 1onic 2023-04-19 Build site.
html e1b57ff 1onic 2023-03-29 Build site.
html 7288d9c 1onic 2023-03-29 Build site.
html 3aef7a8 1onic 2023-03-29 Build site.
html 3f796b6 1onic 2023-03-22 Build site.
html cdae3ea 1onic 2023-03-10 Build site.
html 99420b0 1onic 2023-03-10 Build site.
html 0f70ffe 1onic 2023-03-10 Build site.
html e4922c1 1onic 2023-03-02 Build site.
html 2d991e9 1onic 2023-03-02 Build site.
html c350c44 1onic 2023-02-06 Build site.
html f12e92e 1onic 2023-02-06 Build site.
html 0b2cc20 1onic 2023-02-06 Build site.
html fdd0022 1onic 2023-02-06 Build site.
html 21d42f1 1onic 2023-01-27 Build site.
html e47cac4 1onic 2023-01-27 Build site.
html bce6cff 1onic 2023-01-11 Build site.
html c51b055 1onic 2023-01-11 Build site.
html baf3f16 1onic 2023-01-11 Build site.
html f1c0cc6 1onic 2023-01-11 Build site.
html cbbb209 1onic 2022-12-09 Build site.
html fec1e8a 1onic 2022-12-09 Build site.
html 5006f39 1onic 2022-08-04 Build site.
html a6f860c 1onic 2022-08-04 Build site.
html 57e862a 1onic 2022-08-04 Build site.
html d9c1a09 1onic 2022-08-04 Build site.
html 8a38159 1onic 2022-08-04 Build site.
html 12331d2 1onic 2022-08-04 Build site.
html c814105 1onic 2022-08-04 Build site.
html c92e0bf 1onic 2022-08-03 Build site.
html 16b143f 1onic 2022-07-20 Build site.
html 3393b21 1onic 2022-07-20 Build site.
html a0de791 1onic 2022-07-20 Build site.
html bdf7cc6 1onic 2022-07-20 Build site.
html 02d90b5 1onic 2022-07-20 Build site.
html 0c1180e 1onic 2022-07-20 Build site.
html 52e007d 1onic 2022-07-20 Build site.
html 1b61926 1onic 2022-07-20 Build site.
html fd6de4c 1onic 2022-07-20 Build site.
html 635a240 1onic 2022-07-15 Build site.
html d13dab9 1onic 2022-07-13 Build site.
html 695caed 1onic 2022-07-13 Build site.
html ff7c8b1 1onic 2022-07-13 Build site.
html a8c96b0 1onic 2022-07-07 Build site.
html b06249e 1onic 2022-07-07 Build site.
html 04de0ae 1onic 2022-06-29 Build site.
html 2da0805 1onic 2022-06-29 Build site.
html 5eee051 1onic 2022-06-29 Build site.
html 1516b6d 1onic 2022-06-29 Build site.
html 28a9475 1onic 2022-06-28 Build site.
html 58646b4 1onic 2022-06-15 Build site.
html f4b1305 1onic 2022-06-15 Build site.
html cc4be00 N 2022-05-18 Build site.
html 1401122 N 2022-05-18 Build site.
html 557a2a9 N 2022-05-18 Build site.
html c41d4f9 N 2022-05-12 Build site.
html 1261222 N 2022-05-12 Build site.
html 90df802 N 2022-05-11 Build site.
html 6e985df N 2022-05-11 Build site.
html a844654 N 2022-05-11 Build site.
html aa46898 N 2022-05-11 Build site.
html d79e35c N 2022-05-11 Build site.
html bfc20a5 N 2022-05-04 Build site.
html ab1c893 N 2022-05-04 Build site.
html af6c895 N 2022-05-04 Build site.
html 001acfd N 2022-04-29 Build site.
html 52bb457 N 2022-04-29 Build site.
html 947de76 N 2022-04-29 Build site.
html ea30a16 N 2022-04-29 Build site.
html 299e079 N 2022-04-29 Build site.
html 30904e4 N 2022-04-29 Build site.
html 3540605 N 2022-04-29 Build site.
html f681be8 N 2022-04-29 Build site.
html a110a99 N 2022-04-27 Build site.
html 3592f27 N 2022-04-27 Build site.
html 37aa3b8 N 2022-04-27 Build site.
html 5bad46b N 2022-04-27 Build site.
html 7edf2ca N 2022-04-27 Build site.
html f05d18b N 2022-04-21 Build site.
html 976ffc2 N 2022-04-21 Build site.
html 8442797 N 2022-04-21 Build site.
html 695da69 N 2022-04-21 Build site.
html 8e47783 N 2022-04-21 Build site.
html 977ea3e N 2022-04-21 Build site.
html b72fe91 N 2022-04-21 Build site.
html edc54ba N 2022-04-21 Build site.
html dd8b725 N 2022-04-21 Build site.
html e2733c6 N 2022-04-21 Build site.
html 32987ea N 2022-04-21 Build site.
html d1c363f N 2022-04-21 Build site.
html 984b514 N 2022-04-21 Build site.
html a1aa819 N 2022-04-21 Build site.
html 725c775 N 2022-04-21 Build site.
html d31a7f8 N 2022-04-21 Build site.
html 47842f8 N 2022-04-21 Build site.
html c997e70 N 2022-04-21 Build site.
html 2f8b4a5 N 2022-04-21 Build site.
html 3012325 N 2022-04-21 Build site.
html 99fa823 N 2022-04-21 Build site.
html 53e5c0a N 2022-04-21 Build site.
html e2c4450 N 2022-04-21 Build site.
html 67323bc N 2022-04-21 Build site.
html 47fc19a N 2022-04-21 Build site.
html 958dea7 N 2022-04-21 Build site.
html a2b7524 N 2022-04-21 Build site.
html 4b231e9 N 2022-04-21 Build site.
html ea6da48 N 2022-04-21 Build site.
html 56c8aba N 2022-04-21 Build site.
html 649b842 N 2022-04-21 Build site.
html 3a63d56 N 2022-04-20 Build site.
html 6d20288 N 2022-04-20 Build site.
html fd4742b N 2022-04-07 Build site.
html fa53b92 N 2022-04-07 Build site.
html b472f9a N 2022-04-07 Build site.
html 7240cfb N 2022-04-07 Build site.
html ecf9247 N 2022-04-07 Build site.
html e865c25 N 2022-04-07 Build site.
html 3f5f68c N 2022-04-07 Build site.
html dd23b48 N 2022-04-07 Build site.
html dca45ed N 2022-04-07 Build site.
html 2b73c03 N 2022-04-07 Build site.
html 047c6b5 N 2022-04-07 Build site.
html b2d88e7 N 2022-04-07 Build site.
html 32e9594 N 2022-04-07 Build site.
html 65a9a67 N 2022-04-07 Build site.
html 75c5e7a N 2022-04-07 Build site.
html 78c9e97 1onic 2022-03-30 Build site.
Rmd 959b8be 1onic 2022-03-30 update
html dadcb5a 1onic 2022-03-30 Build site.
Rmd d7642da 1onic 2022-03-30 update
html 9451ff4 1onic 2022-03-30 Build site.
Rmd 8751319 1onic 2022-03-30 update
html 5610a89 1onic 2022-03-30 Build site.
html 88fd61f 1onic 2022-03-30 Build site.
Rmd dbb75ae 1onic 2022-03-30 update

Midway3 vs Midway2

The goal of this section is to discuss advantages of CROPseq/SCEPTRE analysis on midway3.

There are two primary advantages to midway3 over midway2 for SCEPTRE analysis: memory and CPUs per node. Midway3s advantage comes from an increase in its node CPUs and memory – both of which are resources currently limiting our trans-wide analysis.

Comparison of Node Resources:

  • Midway2:
    • 28 cores (28 x Intel E5-2680v4 2.4GHz)
    • 64 GB RAM
    • 200-300 USD msrp (3/30/2022)
  • Midway3:
    • 48 cores (2x Intel Xeon Gold 6248R)
    • 128 GB RAM
    • 2000 USD msrp (3/30/2022)

Our current strategy is to break tran-wide SCEPTRE analysis into groups capable of being run within the given resources of midway2 (each group is also subsequently broken into pods which are run parallel). This limits the amount of gene-gRNA pairs we can test before we exhaust the resources on a node (memory) or time (36hr walltime). Currently we are seeing an exhaustion of memory due to the nature of running parallel pods within each group. This occurs because the sparse matrixes (gene matrix is the largest, followed by the gRNA matrix, then covariate matrix) must be loaded into memory multiple times. When large datasets (large gene matrixes) are distributed across 28 cores (midway2 node max) we load each of the 3 primary matrix 3 x CORES times. This becomes an issue if a large dataset must be loaded 28 times. Parallelization has a memory tradeoff.

Midway3 offers an improvement because it both increases available cores (28 > 48), nearly doubling them, and also increases node RAM from a meager 64 GB to 192 GB. This means we can have more parallel pods running within each gene-gRNA group and we can also handle larger datasets because of the memory overhead being increased to a greater degree than core count. For larger datasets in the future we would plan to either use all 48 cores and rely on our memory footprint being smaller than 192 GB limit or, alternatively, with large datasets, we will decrease our core usage down from the maximum until the analysis fits within 192 GB RAM.

The other advantages that midway3 offers that are potentially of great use:

  1. More GPU nodes and larger memory limits for working with extremely large datasets.
  2. SSD storage
  • Its worth noting that Midway2 is hard drive only and SSDs offer greatly increased speeds which will generally decrease computation time accross the board for ALL applications.
  • Each node now has 960 GB of SSD storage