-Dockerized version of the PRISM MTS pipeline. For use with Merino pipeline output or [clue.io](clue.io) datasets.
+# MTS Pipeline (with Docker images)
-To get the associated docker image run:
-```{bash}
-docker pull aboghoss/clue-mts
-```
+Production version of the [PRISM](https://www.theprismlab.org/) MTS pipeline. For use with Luminex output or [clue.io](clue.io) datasets.
-Example command to run the container (see `docker_wrapper.sh`):
-```{bash}
-bash docker_wrapper.sh -i -o -p
-```
+Each module is organized into its own sub-directory with associated Docker images on [Docker Hub](https://hub.docker.com/orgs/prismcmap/repositories). `README` files for each module are contained within their respective directories.
-Requires the following files in the input data folder:
-- PR300 and PR500 Level 2 data (.gctx format)
-- PR300 and PR500 cell_info (long table)
-- PR300 and PR500 inst_info (long table)
-- project_key (long table mapping compounds to projects)
+## Pre-requisites
-All files other than the Level 2 data can be in any format readable by `data.table::fread`
+### Docker
+
+In order to run the Docker images (no R installation or local scripts required), install Docker following the instructions [here](https://docs.docker.com/get-docker/).
+
+### R and RStudio (optional)
+
+To run individual scripts or make modifications install [R](https://www.r-project.org/). [RStudio](https://www.rstudio.com/products/rstudio/), an IDE for R, is not required but is recommended for running and viewing R code.
+
+## Running the MTS Pipeline
+
+The MTS Pipeline can be broken down into three stages (for the purpose of visualization):
+
+1. Generating long form tables of luminescence and log-fold change (LFC) from raw plate map and luminescence (csv) data
+2. Running COMBAT batch-correction on collapsed LFC files
+3. Running dose response curve (drc) fitting, biomarker analysis, and generating interactive reports
+
+
+
+#### Stage 1: Generating long form tables of luminescence and log-fold change (LFC) from raw plate map and luminescence (csv) data
+
+
+
+
+
+Our analytical pipeline processes data from plate maps files, which contain sample metadata such as dose, compound identifiers, and plate locations. In conjunction with these plate maps, we use jcsvs that store bead counts and median fluorescence intensity values for each sample across analytes (representing individual cell lines). This combined information is used to generate files for our various data levels:
+
+
Level 3: log2 mean fluorescence intensity, both raw and normalized (logMFI, logMFI_norm)
+The process for analysis for PR500 and PR300 is the same, however, they map to different cell sets and are therefore run in series.
+
+
+
+#### Stage 2: Running COMBAT batch-correction on collapsed LFC files
+
+
+
+
+
+The PR300 and PR500 build results are then used as inputs to a combined analysis ending in report generation (next figure). After pre-processing, batch correction (COMBAT) is conducted in a pool-wise manner on Level 4 and Level 5 data as specified in [Johnson et al](https://pubmed.ncbi.nlm.nih.gov/16632515/). It is worth noting that for our new Extended PRISM screening (EPS), the initial results for pre-processing are incorporated into the MTS pipeline after batch correction, as denoted by the red star (dose responsive fitting, biomarker analysis, and reports are the same).
+
+
+
+#### Stage 3: Running dose response curve (drc) fitting, biomarker analysis, and generating interactive reports
+
+
+
+
+
+Level 4 data is subsequently fed into the drc module to acquire drc fitting parameters (i.e., AUC and IC50). Following this, drc parameters and Level 5 data are inputted into the biomarker module for univariate and multivariate analysis alongside cell line genomic feature data. Finally, the drc, biomarker, and Level 4/5 data are inputted into the report generation module to create reports, which are then uploaded onto our portal.
+It is important to note that the dose response and biomarker modules may take some time to run depending on your available hardware.
+
+
+
+## A note about running locally
+
+The MTS pipeline is run on AWS and therefore most modules are designed with AWS in mind. Some modules are also specifically for handling and moving around files on AWS. Therefore, while the Docker images are useful tools, it may be easier in some cases to run the R scripts individually when running locally
diff --git a/assemble/.dockerignore b/assemble/.dockerignore
new file mode 100644
index 0000000..dfd6481
--- /dev/null
+++ b/assemble/.dockerignore
@@ -0,0 +1 @@
+test_*.py
diff --git a/assemble/Dockerfile b/assemble/Dockerfile
new file mode 100644
index 0000000..a8d8ad1
--- /dev/null
+++ b/assemble/Dockerfile
@@ -0,0 +1,35 @@
+FROM prismcmap/base-prism:latest
+MAINTAINER Anup Jonchhe
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+#RUN cd /cmap && \
+#conda create -y --name assemble -c bioconda python=3.6 numpy pandas yaml h5py requests setuptools argparse pathlib yaml pyyaml jinja2 && \
+#git clone https://github.com/cmap/cmapPy.git && \
+#cd /
+#
+#SHELL ["conda", "run", "-n", "assemble", "/bin/bash", "-c"]
+#
+#RUN cd /cmap/cmapPy && \
+#python setup.py install && \
+#cd /
+
+
+RUN mkdir -p /clue/bin/assemble/utils/
+
+COPY ./assemble.py /clue/bin/assemble/assemble.py
+COPY ./setup_logger.py /clue/bin/assemble/setup_logger.py
+COPY ./utils/exceptions.py /clue/bin/assemble/utils/exceptions.py
+COPY ./utils/path_utils.py /clue/bin/assemble/utils/path_utils.py
+COPY ./assemble_core.py /clue/bin/assemble/assemble_core.py
+COPY ./batch_assemble.sh /clue/bin/assemble/batch_assemble
+COPY ./davepool_data.py /clue/bin/assemble/davepool_data.py
+COPY ./parse_data.py /clue/bin/assemble/parse_data.py
+COPY ./prism_metadata.py /clue/bin/assemble/prism_metadata.py
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+WORKDIR /
+
+ENV PATH /clue/bin/assemble/:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["batch_assemble"]
diff --git a/assemble/README.md b/assemble/README.md
new file mode 100644
index 0000000..04ef941
--- /dev/null
+++ b/assemble/README.md
@@ -0,0 +1,52 @@
+# Assemble
+
+## Assemble overview
+
+Conversion from LXB files from Luminex bead scanners to measured fluorescent intensity (MFI) values
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/merino) run:
+
+```
+docker pull prismcmap/merino:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+Help text is available using the command:
+
+```
+docker run prismcmap/merino --help
+```
+
+The entrypoint for the Docker is `batch_assemble.sh` which is a wrapper around `assemble.py`. It can be inspected to see how the environment within the Docker container is set up
+
+### Example usage with Docker
+
+Docker is the recommended method for running the assemble module as it uses a deprecated version of python (Python 2.7).
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+
+```
+docker run \
+ -it \
+ -v ~/TEST:/data/ \
+ prismcmap/merino:latest \
+ -csv_filepath /data/lxb/TEST005_PR500_120H_X1_P6/TEST005_PR500_120H_X1_P6.jcsv \
+ -plate_map_path /data/map_src/TEST005.src \
+ -out ~/TEST/build/ -assay_type PR500.CS5.3
+```
+
+### Usage with Python
+
+While running using Docker is the preferred method, it is possible to set up a conda environment with the required versions of python and other packages.
+
+More information can be found in the [merino Github Repo](https://github.com/cmap/merino)
+
+
+
+
diff --git a/assemble/__init__.py b/assemble/__init__.py
new file mode 100644
index 0000000..bd43980
--- /dev/null
+++ b/assemble/__init__.py
@@ -0,0 +1,4 @@
+import os.path
+
+#default_config_filepath = os.path.expanduser("~/.prism_pipeline.cfg")
+default_config_filepath = "https://s3.amazonaws.com/analysis.clue.io/vdb/merino/prism_pipeline.cfg"
diff --git a/assemble/__pycache__/assemble_core.cpython-38.pyc b/assemble/__pycache__/assemble_core.cpython-38.pyc
new file mode 100644
index 0000000..5d8a371
Binary files /dev/null and b/assemble/__pycache__/assemble_core.cpython-38.pyc differ
diff --git a/assemble/assemble.py b/assemble/assemble.py
new file mode 100644
index 0000000..bc316ee
--- /dev/null
+++ b/assemble/assemble.py
@@ -0,0 +1,222 @@
+"""
+
+Command line script which takes the two CSVs belonging to a single PRISM replicate and combines them into a GCT file
+along with all of the relevant meta data.
+
+The meta data inputs are a plate map, a cell set definition file, a plate tracking file, and a davepool-analyte mapping.
+"""
+import os
+import sys
+import ast
+import json
+import yaml
+import logging
+import argparse
+import traceback
+
+
+
+import requests
+import configparser
+from urllib.request import urlopen
+
+import setup_logger as setup_logger
+import utils.exceptions as assemble_exception
+import utils.path_utils as path_utils
+
+import davepool_data as davepool_data
+import prism_metadata as prism_metadata
+import assemble_core as assemble_core
+
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+_prism_cell_config_file_section = "PrismCell column headers"
+_davepool_analyte_mapping_file_section = "DavepoolAnalyteMapping column headers"
+
+pert_column_metadata_fields = ['pert_id', 'pert_dose', 'pert_type', 'pert_well', 'pert_dose_unit',
+ 'pert_iname']
+prismcell_row_metadata_fields = ['ccle_name', 'davepool_id', 'analyte_id', 'barcode_id', 'cell_iname', 'pool_id']
+
+API_URL = 'https://api.clue.io/api/'
+DEV_API_URL = 'https://dev-api.clue.io/api/'
+API_KEY = os.environ['API_KEY']
+default_config_filepath = "https://s3.amazonaws.com/analysis.clue.io/vdb/merino/prism_pipeline.cfg"
+
+def build_parser():
+
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ # The following arguments are required. These are files that are necessary for assembly and which change
+ # frequently between cohorts, replicates, etc.
+ parser.add_argument("-config_filepath", "-cfg", help="path to the location of the configuration file", type=str,
+ default=default_config_filepath)
+ parser.add_argument("-csv_filepath", "-csv", help="full path to csv", type=str, required=True)
+ parser.add_argument("-assay_type", "-at", help="assay data was profiled in",
+ type=str, required=False)
+ parser.add_argument("-beadset", "-bs", help="Bead set used in detection",
+ type=str, default=None, required=False)
+ parser.add_argument("-plate_map_path", "-pmp",
+ help="path to file containing plate map describing perturbagens used", type=str, required=False)
+ parser.add_argument("-map_src_plate", "-map",
+ help="Pert Plate with replicate map name. Searches database, using API KEY environment variable",
+ type=str, required=False)
+
+
+ # These arguments are optional. Some may be superfluous now and might be removed.
+ parser.add_argument("-verbose", '-v', help="Whether to print a bunch of output", action="store_true", default=False)
+ parser.add_argument("--dev", help=argparse.SUPPRESS, action="store_true", default=False)
+
+ parser.add_argument("-cell_set_definition_file", "-csdf",
+ help="file containing cell set definition to use, overriding config file",
+ type=str, default=None, required=False)
+ parser.add_argument("-outfile", "-out", help="location to write gct", type=str,
+ default='')
+ parser.add_argument("-truncate_to_plate_map", "-trunc", help="True or false, if true truncate data to fit framework of platemap provided",
+ action="store_true", default=True)
+
+ return parser
+
+def write_args_to_file(args, outfile):
+ with open(outfile, "w") as f:
+ print(vars(args))
+ yaml.dump(vars(args), f)
+
+def read_csv(csv_filepath, assay_type):
+
+ pd = davepool_data.read_data(csv_filepath)
+ pd.davepool_id = assay_type
+ return pd
+
+def check_for_duplicate_analytes(prism_cell_list):
+ seen_analytes = set()
+ duplicated_analytes = set()
+ for i in range(0,len(prism_cell_list)):
+ analyte_id = prism_cell_list[i].analyte_id
+ if analyte_id in seen_analytes:
+ duplicated_analytes.add(analyte_id)
+ else:
+ seen_analytes.add(analyte_id)
+ if duplicated_analytes:
+ msg = f"The following analytes are duplicated: {', '.join(duplicated_analytes)}, check cell set database."
+ logger.error(msg)
+ raise KeyError(msg)
+
+'''
+There are some cases in which we are subsetting plates into different groups, ie. more than one gct per plate.
+This was the case for PPCN. As such, we need a function to truncate the data to match the plate map which is given.
+:param davepool_data_objects:
+:param all_perturbagens:
+:param truncate_to_platemap:
+:return:
+'''
+def truncate_data_objects_to_plate_map(davepool_data_objects, all_perturbagens, truncate_to_platemap):
+ platemap_well_list = set([p.pert_well for p in all_perturbagens])
+ for davepool in davepool_data_objects:
+ if platemap_well_list == set(davepool.median_data.keys()):
+ return davepool_data_objects
+ elif truncate_to_platemap == True:
+ for d in list(davepool_data_objects[0].median_data):
+ if d not in platemap_well_list:
+ del davepool_data_objects[0].median_data[d]
+
+ for c in list(davepool_data_objects[0].count_data):
+ if c not in platemap_well_list:
+ del davepool_data_objects[0].count_data[c]
+
+ else:
+ msg = "Assemble truncate data objects to plate map: Well lists of platemap and csv do not match"
+ raise assemble_exception.DataMappingMismatch(msg)
+
+ return davepool_data_objects
+
+def main(args, all_perturbagens=None, assay_plates=None):
+
+ prism_replicate_name = os.path.basename(args.csv_filepath).rsplit(".", 1)[0]
+ (_, assay, tp, replicate_number, bead) = prism_replicate_name.rsplit("_")
+
+ if args.beadset:
+ beadset = args.beadset
+ else:
+ beadset = bead
+
+
+ if beadset is not None and args.assay_type is None:
+ api_call = os.path.join('https://api.clue.io/api', 'beadset', beadset)
+ db_entry = requests.get(api_call)
+ args.assay_type = json.loads(db_entry.text)['assay_variant']
+
+ if args.assay_type == None:
+ msg = "No assay type found from beadset - must be specified in arg -assay_type"
+ raise assemble_exception.NoAssayTypeFound(msg)
+
+
+ davepool_id_csv_list = args.csv_filepath
+ davepool_data_objects = [read_csv(davepool_id_csv_list, args.assay_type)]
+
+ # Set up output directory
+ if not os.path.exists(os.path.join(args.outfile, "assemble", prism_replicate_name)):
+ os.makedirs(os.path.join(args.outfile, "assemble", prism_replicate_name))
+
+ # Write args used to yaml file
+ write_args_to_file(args, os.path.join(args.outfile, "assemble", prism_replicate_name, 'config.yaml'))
+
+ #Select API TODO: move to config file
+ api_url = DEV_API_URL if args.dev else API_URL
+
+ if args.map_src_plate is not None:
+ all_perturbagens = prism_metadata.build_perturbagens_from_db(args.map_src_plate, tp, api_url=api_url) # Read from DB
+ elif args.plate_map_path is not None:
+ all_perturbagens = prism_metadata.build_perturbagens_from_file(args.plate_map_path, tp)
+ else:
+ raise ValueError("One of -plate_map_path or -map_src_plate is required")
+
+ for pert in all_perturbagens:
+ pert.validate_properties(pert_column_metadata_fields)
+
+ #read actual data from relevant csv files, associate it with davepool ID
+ prism_cell_list = prism_metadata.build_prism_cell_list_from_db(args.assay_type, beadset, api_url=api_url)
+
+ # check if any analytes are duplicated and if so raise an exception
+ check_for_duplicate_analytes(prism_cell_list)
+
+ #prism_cell_list = prism_metadata.build_prism_cell_list(cp, cell_set_file)
+
+ logger.info("len(prism_cell_list): {}".format(len(prism_cell_list)))
+
+ #expected_prism_cell_metadata_fields = prismcell_row_metadata_fields
+ for cell in prism_cell_list:
+ cell.validate_properties(prismcell_row_metadata_fields)
+
+ # truncate csv to plate map size if indicated by args.truncate_to_plate_map
+ truncate_data_objects_to_plate_map(davepool_data_objects, all_perturbagens, args.truncate_to_plate_map)
+
+ # Pass python objects to the core assembly module (this is where command line and automated assembly intersect)
+ # here the outfile for automation is defined as project_dir/prism_replicate_set_name
+ try:
+ assemble_core.main(prism_replicate_name, args.outfile, all_perturbagens, davepool_data_objects, prism_cell_list, verbose=args.verbose)
+
+ except Exception as e:
+ failure_path = os.path.join(args.outfile, "assemble", prism_replicate_name, "failure.txt")
+ ex_type, ex, tb = sys.exc_info()
+ print("plate {} failed for reason: {}: {}".format(prism_replicate_name, ex_type, ex))
+ traceback.print_tb(tb)
+ with open(failure_path, "w") as file:
+ file.write("plate {} failed for reason: {}: {}\n".format(prism_replicate_name, ex_type, ex))
+ file.write("\ntraceback:\n")
+ traceback.print_tb(tb, file=file)
+ sys.exit(-1)
+
+ success_path = os.path.join(args.outfile, "assemble", prism_replicate_name, "success.txt")
+ with open(success_path, "w") as file:
+ file.write("plate {} successfully assembled".format(prism_replicate_name))
+
+if __name__ == "__main__":
+ args = build_parser().parse_args(sys.argv[1:])
+ setup_logger.setup(verbose=args.verbose)
+
+ logger.info("args: {}".format(args))
+
+ if not (args.map_src_plate or args.plate_map_path):
+ raise ValueError("One of -plate_map_path or -map_src_plate is required")
+
+ main(args)
diff --git a/assemble/assemble_core.py b/assemble/assemble_core.py
new file mode 100644
index 0000000..00ebc92
--- /dev/null
+++ b/assemble/assemble_core.py
@@ -0,0 +1,314 @@
+import setup_logger as setup_logger
+import logging
+import os
+import cmapPy.pandasGEXpress.GCToo as GCToo
+import prism_metadata
+import pandas
+import pandas as pd
+import cmapPy.pandasGEXpress.write_gct as write_gct
+import numpy as np
+from math import floor, log10
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+_remove_row_annotations = ["feature_id", "ignore"]
+_remove_col_annotations = ["assay_plate_barcode"]
+
+_null = "-666"
+_NaN = "NaN"
+
+
+class DataByCell:
+ def __init__(self, cell_data_map=None, well_list=None):
+ self.cell_data_map = cell_data_map
+ self.well_list = well_list
+
+ def __str__(self):
+ return "cell_data_map: {} well_list: {}".format(self.cell_data_map, self.well_list)
+
+
+def build_davepool_id_to_cells_map(prism_cell_list):
+ '''
+ build one-to-many mapping between davepool ID and the corresponding PRISM cell lines that are within that davepool
+ :param prism_cell_list:
+ :return:
+ '''
+ r = {}
+ for pc in prism_cell_list:
+ if pc.davepool_id not in r:
+ r[pc.davepool_id] = []
+ r[pc.davepool_id].append(pc)
+
+ return r
+
+
+def build_data_by_cell(cells, davepool_data_obj):
+ cell_to_median_data_map = {}
+ median_wells = []
+ cell_to_count_data_map = {}
+ count_wells = []
+
+ ld = [(cell_to_median_data_map, median_wells, davepool_data_obj.median_headers, davepool_data_obj.median_data),
+ (cell_to_count_data_map, count_wells, davepool_data_obj.count_headers, davepool_data_obj.count_data)]
+
+ for (cell_data_map, wells, headers, data) in ld:
+
+ cell_header_map = {}
+ for c in cells:
+ if c.ignore == False:
+ analyte_id = str(c.analyte_id).capitalize()
+
+ cell_header_map[c] = headers.index(analyte_id)
+ cell_data_map[c] = []
+
+ for d in data.keys():
+ wells.append(d)
+ for c in cells:
+ if c in cell_header_map:
+ datum_index = cell_header_map[c]
+ datum = data[d][datum_index - 1]
+ else:
+ datum = float('nan')
+
+ cell_data_map[c].append(datum)
+
+ return (DataByCell(cell_to_median_data_map, median_wells), DataByCell(cell_to_count_data_map, count_wells))
+
+
+def combine_maps_with_checks(source_map, dest_map):
+ source_keys = set(source_map.keys())
+ dest_keys = set(dest_map.keys())
+
+ common_keys = source_keys & dest_keys
+
+ if len(common_keys) > 0:
+ msg = "the source_map and dest_map had common_keys: {}".format(common_keys)
+ logger.error(msg)
+ raise Exception("assemble combine_maps_with_checks " + msg)
+ else:
+ dest_map.update(source_map)
+
+
+def process_data(davepool_data_objects, davepool_id_to_cells_map):
+ logger.debug("davepool_id_to_cells_map.keys(): {}".format(davepool_id_to_cells_map.keys()))
+
+ authoritative_well_list = []
+ all_median_data_by_cell = DataByCell(cell_data_map={})
+ all_count_data_by_cell = DataByCell(cell_data_map={})
+ for dd in davepool_data_objects:
+ cells = davepool_id_to_cells_map[dd.davepool_id]
+ logger.debug("pools: {}".format(cells))
+
+ (median_data_by_cell), (count_data_by_cell) = build_data_by_cell(cells, dd)
+
+ if len(authoritative_well_list) == 0:
+ authoritative_well_list = median_data_by_cell.well_list
+ else:
+ assert authoritative_well_list == median_data_by_cell.well_list, (authoritative_well_list,
+ median_data_by_cell.well_list)
+ assert authoritative_well_list == count_data_by_cell.well_list, (authoritative_well_list,
+ count_data_by_cell.well_list)
+
+ combine_maps_with_checks(median_data_by_cell.cell_data_map, all_median_data_by_cell.cell_data_map)
+ combine_maps_with_checks(count_data_by_cell.cell_data_map, all_count_data_by_cell.cell_data_map)
+
+ all_median_data_by_cell.well_list = authoritative_well_list
+ all_count_data_by_cell.well_list = authoritative_well_list
+
+ return (all_median_data_by_cell, all_count_data_by_cell)
+
+
+def build_gctoo(prism_replicate_name, perturbagen_list, data_by_cell):
+ # build column metadata dataframe:
+ def column_ID_builder(perturbagen):
+ return prism_replicate_name + ":" + perturbagen.pert_well
+
+ col_metadata_df = prism_metadata.convert_objects_to_metadata_df(column_ID_builder, perturbagen_list, None)
+
+ for col_annot in _remove_col_annotations:
+ if col_annot in col_metadata_df.columns:
+ col_metadata_df.drop(col_annot, axis=1, inplace=True)
+
+ col_metadata_df.sort_index(inplace=True)
+ col_metadata_df['prism_replicate'] = prism_replicate_name
+ col_metadata_df['data_level'] = 'assemble'
+ col_metadata_df['provenance'] = 'assembled'
+ logger.info("my_gctoo.col_metadata_df.shape: {}".format(col_metadata_df.shape))
+ logger.debug("my_gctoo.col_metadata_df: {}".format(col_metadata_df))
+
+ ########################################
+
+ # build row metadata dataframe:
+ def row_ID_builder(prism_cell_obj):
+ return prism_cell_obj.feature_id
+
+ row_metadata_df = prism_metadata.convert_objects_to_metadata_df(row_ID_builder,
+ data_by_cell.cell_data_map.keys(), {})
+
+ for row_annot in _remove_row_annotations:
+ if row_annot in row_metadata_df.columns:
+ row_metadata_df.drop(row_annot, axis=1, inplace=True)
+
+ row_metadata_df.sort_index(inplace=True)
+ logger.info("my_gctoo.row_metadata_df.shape: {}".format(row_metadata_df.shape))
+ logger.debug("my_gctoo.row_metadata_df: {}".format(row_metadata_df))
+ ########################################
+
+ # build data dataframe - will have the cells as columns and rows as wells, and then transpose
+ # start by building mapping between cell ID and corresponding data for that cell, to populate dataframe columns
+ cell_id_data_map = {}
+ for (c, data) in data_by_cell.cell_data_map.items():
+ id = row_ID_builder(c)
+ cell_id_data_map[id] = data
+ logger.debug("cell_id_data_map: {}".format(cell_id_data_map))
+
+ # build index for the rows of the dataframe using what is actually the column ID (since it will be transposed)
+ well_perturbagen_map = {}
+ for p in perturbagen_list:
+ well_perturbagen_map[p.pert_well] = p
+ data_df_column_ids = []
+ for w in data_by_cell.well_list:
+ p = well_perturbagen_map[w]
+ data_df_column_ids.append(column_ID_builder(p))
+
+ data_df = build_gctoo_data_df(cell_id_data_map, data_df_column_ids)
+ ########################################
+ my_gctoo = GCToo.GCToo(data_df=data_df, row_metadata_df=row_metadata_df, col_metadata_df=col_metadata_df)
+
+ return my_gctoo
+
+
+def build_gctoo_data_df(cell_id_data_map, data_df_column_ids):
+ '''
+ build the pandas dataframe that will be used for the data_df part of a gctoo object
+ :param cell_id_data_map:
+ :param data_df_column_ids:
+ :return:
+ '''
+ data_df = pandas.DataFrame(cell_id_data_map, index=data_df_column_ids).T
+ data_df.sort_index(axis=0, inplace=True)
+ data_df.sort_index(axis=1, inplace=True)
+ data_df.replace("", value=_NaN, inplace=True)
+ logger.info("data_df.shape: {}".format(data_df.shape))
+ logger.debug("data_df: {}".format(data_df))
+
+ return data_df
+
+
+"""
+stringify method to write floats as numerical non-scientific notation
+"""
+def float_to_str(f):
+ float_string = repr(f)
+ if 'e' in float_string: # detect scientific notation
+ digits, exp = float_string.split('e')
+ digits = digits.replace('.', '').replace('-', '')
+ exp = int(exp)
+ zero_padding = '0' * (abs(int(exp)) - 1) # minus 1 for decimal point in the sci notation
+ sign = '-' if f < 0 else ''
+ if exp > 0:
+ float_string = '{}{}{}.0'.format(sign, digits, zero_padding)
+ else:
+ float_string = '{}0.{}{}'.format(sign, zero_padding, digits)
+ return float_string
+
+
+"""
+rounds to significant figures
+"""
+def _round_sig(x, sig=5):
+ if x == 0:
+ return 0
+ else:
+ return round(x, sig - int(floor(log10(abs(x)))) - 1)
+
+
+"""
+prints string as decimal value not scientific notation
+"""
+def _format_floats(fl, sig=4, max_precision=50):
+ if type(fl) == str:
+ fl = float(fl)
+ if (fl is None) or np.isnan(fl):
+ return np.nan
+ else:
+ return float_to_str(round(_round_sig(fl, sig=sig), max_precision))
+
+
+def process_pert_doses(el):
+ logger.debug("el: {}".format(el))
+ if pd.isna(el) or (el == 'nan'):
+ return el
+ if type(el) == str:
+ el = el.strip()
+ return '|'.join(map(_format_floats, map(float, el.split('|'))))
+ else:
+ return _format_floats(el)
+
+
+def process_pert_idoses(el):
+ logger.debug("el: {}".format(el))
+ if pd.isna(el) or (el=='nan') or ~(el):
+ return el
+ if type(el) == str:
+ el = el.strip()
+ idoses = el.split('|')
+ idoses = [i.split(" ") for i in idoses]
+ return "|".join(["{} {}".format(_format_floats(idose[0]), idose[1]) for idose in idoses])
+ else:
+ return _format_floats(el)
+
+def stringify_inst_doses(inst):
+ # cast pert_dose field to str
+ inst['pert_dose'] = inst['pert_dose'].apply(
+ lambda el: process_pert_doses(el)
+ )
+
+ if 'pert_idose' in inst.columns:
+ inst['pert_idose'] = inst['pert_idose'].apply(
+ lambda el: process_pert_idoses(el)
+ )
+
+ inst['pert_dose'] = inst['pert_dose'].astype(str)
+ return inst
+
+
+def main(prism_replicate_name, outfile, all_perturbagens, davepool_data_objects, prism_cell_list, verbose=False):
+ level = (logging.DEBUG if verbose else logging.INFO)
+ logging.basicConfig(level=level)
+ logger.setLevel(level)
+
+ # Build one-to-many mapping between davepool ID and the multiple PRISM cell lines that are within that davepool
+ davepool_id_to_cells_map = build_davepool_id_to_cells_map(prism_cell_list)
+
+ # Put all the data in gct-able form
+ (all_median_data_by_cell, all_count_data_by_cell) = process_data(davepool_data_objects, davepool_id_to_cells_map)
+
+ # Create full outfile, build the gct, and write it out!
+ median_outfile = os.path.join(outfile, "assemble", prism_replicate_name, prism_replicate_name + "_MEDIAN.gct")
+ median_gctoo = build_gctoo(prism_replicate_name, all_perturbagens, all_median_data_by_cell)
+
+
+ # enforce doses as strings
+ try:
+ logger.info("Attempting to convert doses to strings")
+ inst = stringify_inst_doses(median_gctoo.col_metadata_df)
+ except TypeError as e:
+ inst = median_gctoo.col_metadata_df
+ logger.warning("Could not stringify doses due to TypeError: {}".format(e))
+ except Exception as e:
+ logger.error("Could not stringify doses due to ValueError: {}".format(e))
+ exit(-1)
+ median_gctoo.col_metadata_df = inst
+
+ write_gct.write(median_gctoo, median_outfile, data_null=_NaN, filler_null=_null)
+
+ # Write Inst info file
+ instinfo_outfile = os.path.join(outfile, "assemble", prism_replicate_name, prism_replicate_name + "_inst_info.txt")
+ inst.reset_index().rename(columns={'index':'profile_id'}).to_csv(instinfo_outfile, sep='\t', index=False)
+ logger.info("Instinfo has been written to {}".format(instinfo_outfile))
+
+ count_outfile = os.path.join(outfile, "assemble", prism_replicate_name, prism_replicate_name + "_COUNT.gct")
+ count_gctoo = build_gctoo(prism_replicate_name, all_perturbagens, all_count_data_by_cell)
+ count_gctoo.col_metadata_df = inst
+ write_gct.write(count_gctoo, count_outfile, data_null=_NaN, filler_null=_null)
diff --git a/assemble/batch_assemble.sh b/assemble/batch_assemble.sh
new file mode 100755
index 0000000..7cb5221
--- /dev/null
+++ b/assemble/batch_assemble.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+print_help () {
+ source activate prism
+# cd /cmap/merino/
+# python setup.py develop
+ python /clue/bin/assemble/assemble.py --help
+}
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit_code=$?
+ exit $exit_code
+ ;;
+ -config_root|---config_root)
+ shift
+ CONFIG_ROOT="$1"
+ ;;
+ -project_code|--project_code)
+ shift
+ PROJECT_CODE="$1"
+ ;;
+ -replicate_map|--replicate_map)
+ shift # past argument
+ REPLICATE_MAP="$1"
+ ;;
+ -assay_type|--assay_type)
+ shift # past argument
+ ASSAY_TYPE="$1"
+ ;;
+ -beadset|--beadset)
+ shift # past argument
+ BEADSET="$1"
+ ;;
+ --default)
+ DEFAULT=YES
+ ;;
+ --dev)
+ DEV=TRUE
+ ;;
+ --verbose|-v)
+ VERBOSE=TRUE
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift # past argument or value
+done
+
+source activate prism
+
+python --version
+
+echo CONFIG_ROOT = "${CONFIG_ROOT}"
+echo PROJECT_CODE = "${PROJECT_CODE}"
+echo REPLICATE_MAP = "${REPLICATE_MAP}"
+echo ASSAY_TYPE = "${ASSAY_TYPE}"
+
+IFS=',' read -r -a plates <<< "${PLATES}"
+
+batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+PLATE="${plates[${batch_index}]}"
+echo "PLATE IS: ${PLATE}"
+
+IFS='_' read -r -a plate_token <<< "${PLATE}";
+
+IFS='.' read -r -a replicate_token <<< "${plate_token[3]}";
+
+MAP_SRC_NAME=${plate_token[0]}.${replicate_token[0]}
+
+if [ "${REPLICATE_MAP}" = "TRUE" ];
+then
+ PLATE_MAP_PATH="${CONFIG_ROOT}${PROJECT_CODE}/map_src/${plate_token[0]}.${plate_token[3]}.src"
+else
+ PLATE_MAP_PATH="${CONFIG_ROOT}${PROJECT_CODE}/map_src/${plate_token[0]}.src"
+fi
+
+echo PLATE_MAP_PATH = "${PLATE_MAP_PATH}"
+OUTFILE="${CONFIG_ROOT}${PROJECT_CODE}/${plate_token[0]}_${plate_token[1]}_${plate_token[2]}"
+
+echo OUTFILE = "${OUTFILE}"
+# Activate conda environment
+
+CSV_FILEPATH="${CONFIG_ROOT}${PROJECT_CODE}/lxb/${PLATE}/${PLATE}.jcsv"
+echo CSV_FILEPATH = "${CSV_FILEPATH}"
+
+args=(
+ -map "${MAP_SRC_NAME} "
+ -csv "${CSV_FILEPATH}"
+ -out "${OUTFILE}"
+ -assay_type ${ASSAY_TYPE}
+)
+
+if [[ -n $DEV ]]
+then
+ args+=(--dev)
+fi
+
+if [[ -n $BEADSET ]]
+then
+ args+=(--beadset $BEADSET)
+fi
+
+if [[ -n $VERBOSE ]]
+then
+ args+=(-verbose)
+fi
+
+echo python /clue/bin/assemble/assemble.py "${args[@]}"
+python /clue/bin/assemble/assemble.py "${args[@]}"
+exit_code=$?
+
+# Deactivate conda environment
+source deactivate
+exit $exit_code
+#
diff --git a/assemble/build_docker.sh b/assemble/build_docker.sh
new file mode 100755
index 0000000..260a293
--- /dev/null
+++ b/assemble/build_docker.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+REPO="prismcmap"
+NAME="assemble"
+
+#VERSION="v1.0.2" #change the version number for each new build
+TAG="testing"
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t $REPO/$NAME:$TAG --rm=true .
+#
+#docker push $REPO/$NAME:$VERSION
+docker push $REPO/$NAME:$TAG
+
+#v1.0.2 - changed base Dockerfile to include conda env
+
+# trigger
\ No newline at end of file
diff --git a/assemble/davepool_data.py b/assemble/davepool_data.py
new file mode 100644
index 0000000..7ea8e2f
--- /dev/null
+++ b/assemble/davepool_data.py
@@ -0,0 +1,139 @@
+import logging
+import csv
+
+import setup_logger as setup_logger
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+datatype_names = ["Median", "Count"]
+date_header = "Date"
+datatype_header = "DataType:"
+
+
+class DavepoolData(object):
+ '''
+ contains the data associated with a davepool - davepool id, csv file information,
+ relevant data read from the csv
+ '''
+ def __init__(self, csv_filepath=None, csv_datetime=None, median_headers=None, median_data=None,
+ count_headers=None, count_data=None, davepool_id=None):
+
+ self.csv_filepath = csv_filepath
+ self.csv_datetime = csv_datetime
+ self.median_headers = median_headers
+ self.median_data = median_data
+ self.count_headers = count_headers
+ self.count_data = count_data
+ self.davepool_id = davepool_id
+
+ def __repr__(self):
+ return " ".join(["{}:{}".format(k,v) for (k,v) in self.__dict__.items()])
+
+ def validate_data(self):
+ data_arrays = [("median", self.median_data), ("count", self.count_data)]
+ for (data_name, da) in data_arrays:
+ for (i, md) in enumerate(da):
+ assert len(md) > 0, "data row is empty - self.davepool_id: {} self.csv_filepath: {} data_name: {} row number i: {} previous row - self.median_data[i-1]: {}".format(
+ self.davepool_id, self.csv_filepath, data_name, i, self.median_data[i-1])
+
+def get_datatype_range(data, datatype_names):
+ datatype_indexes = [i for (i,row) in enumerate(data) if len(row) > 0 and row[0] == datatype_header]
+ datatypes = [data[i][1] for i in datatype_indexes]
+ i = len(data) - 1
+ while i > datatype_indexes[-1] and len(data[i]) == 0:
+ i = i - 1
+ datatype_indexes.append(i + 2)
+
+ r = {}
+ for dn in datatype_names:
+ if dn in datatypes:
+ dn_index = datatypes.index(dn)
+ r[dn] = (datatype_indexes[dn_index], datatype_indexes[dn_index+1])
+ else:
+ r[dn] = (None, None)
+
+ return r
+
+
+def get_datetime_from_header_rows(header_rows, csv_filepath):
+ datetime_row = [x for x in header_rows if len(x) > 0 and x[0] == date_header]
+ assert len(datetime_row) == 1, "expected to find only one datetime_row, found datetime_row: {} csv_filepath: {}".format(
+ datetime_row, csv_filepath)
+
+ datetime_row = datetime_row[0]
+
+ r = " ".join(datetime_row[1:]) if len(datetime_row) > 1 else None
+
+ return r
+
+
+def read_data(csv_filepath):
+ f = open(csv_filepath, 'rU')
+ reader = csv.reader(f)
+
+ data = list()
+ for row in reader:
+ data.append(row)
+ f.close()
+
+ logger.debug("len(data): {}".format(len(data)))
+
+ datatype_ranges = get_datatype_range(data, datatype_names)
+
+ datetime = get_datetime_from_header_rows(data[0:datatype_ranges[datatype_names[0]][0]], csv_filepath)
+
+ for dn in datatype_names:
+ if datatype_ranges[dn][0] is None:
+ raise Exception("davepool_data read_data did not find expected DataType dn: {}".format(dn))
+
+ median_range = datatype_ranges[datatype_names[0]]
+ count_range = datatype_ranges[datatype_names[1]]
+
+ pd = DavepoolData(csv_filepath=csv_filepath, csv_datetime=datetime)
+ pd.median_headers = data[median_range[0]+1]
+ logger.info("len(pd.median_headers): {}".format(len(pd.median_headers)))
+
+ pd.median_data = {}
+ pd.count_data = {}
+
+ median_data = data[(median_range[0]+2):(median_range[1]-1)]
+
+ for well in median_data:
+ if len(well) < len(pd.median_headers):
+ well.extend(['Nan'] * (len(pd.median_headers) - len(well)))
+ pd.median_data[parse_location_to_well(well[0])] = well[1:]
+ else:
+ pd.median_data[parse_location_to_well(well[0])] = well[1:]
+
+ logger.info("len(pd.median_data): {}".format(len(pd.median_data)))
+ logger.debug("first line of median data - pd.median_data[0]: {}".format(pd.median_data['A01']))
+ logger.debug("last line of median data - pd.median_data[-1]: {}".format(pd.median_data['P24']))
+
+ pd.count_headers = data[count_range[0]+1]
+ logger.info("len(pd.count_headers): {}".format(len(pd.count_headers)))
+
+ count_data = data[(count_range[0]+2):(count_range[1]-1)]
+
+ for well in count_data:
+ if len(well) < len(pd.count_headers):
+ well.extend(['Nan'] * (len(pd.count_headers) - len(well)))
+ pd.count_data[parse_location_to_well(well[0])] = well[1:]
+ else:
+ pd.count_data[parse_location_to_well(well[0])] = well[1:]
+
+ logger.info("len(pd.count_data): {}".format(len(pd.count_data)))
+ logger.debug("first line of count data - pd.count_data[0]: {}".format(pd.count_data['A01']))
+ logger.debug("last line of count data - pd.count_data[-1]: {}".format(pd.count_data['P24']))
+
+ pd.validate_data()
+
+ return pd
+
+def parse_location_to_well(location):
+ split = location.split(",")
+ right_paren_index = split[1].index(")")
+ raw_well = split[1][0:right_paren_index]
+ row = raw_well[0]
+ raw_col = raw_well[1:]
+ col = raw_col.zfill(2)
+ return row + col
diff --git a/assemble/docker-base/Dockerfile b/assemble/docker-base/Dockerfile
new file mode 100644
index 0000000..97a4b07
--- /dev/null
+++ b/assemble/docker-base/Dockerfile
@@ -0,0 +1,21 @@
+FROM continuumio/miniconda
+MAINTAINER Jacob Asiedu
+LABEL base.merino.pipeline.clue.io.version="0.0.1"
+LABEL base.merino.pipeline.clue.io.vendor="Connectivity Map"
+RUN mkdir -p /cmap/bin && \
+mkdir -p /cmap/merino/ && \
+cd /cmap/merino/
+#conda create -y --name merino -c bioconda python=2.7.11 numpy pandas h5py=2.6.0 requests=2.13.0 cmapPy=3.2.0 scipy setuptools argparse=1.4.0 boto3 pathlib patsy matplotlib seaborn statsmodels yaml pyyaml jinja2
+
+RUN which python
+
+RUN cd /cmap && \
+conda create -y --name prism -c bioconda python=3.6 numpy pandas yaml h5py requests setuptools argparse pathlib yaml pyyaml jinja2 simplejson && \
+git clone https://github.com/cmap/cmapPy.git && \
+cd /
+
+SHELL ["conda", "run", "-n", "prism", "/bin/bash", "-c"]
+
+RUN cd /cmap/cmapPy && \
+python setup.py install && \
+cd /
diff --git a/assemble/docker-base/docker_build.sh b/assemble/docker-base/docker_build.sh
new file mode 100644
index 0000000..618e516
--- /dev/null
+++ b/assemble/docker-base/docker_build.sh
@@ -0,0 +1 @@
+docker build --platform linux/amd64 -t prismcmap/base-prism:latest -t prismcmap/base-prism:v0.0.3 --rm=true .
diff --git a/assemble/local.cfg b/assemble/local.cfg
new file mode 100644
index 0000000..059a9fb
--- /dev/null
+++ b/assemble/local.cfg
@@ -0,0 +1,69 @@
+[CS12.1]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_CS12.1_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/CS12.1_mapping.txt
+
+##########################################
+[CS12.2]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_CS12.2_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/CS12.2_mapping.txt
+
+##########################################
+[CS12.3]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_CS12.3_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/CS12.3_mapping.txt
+
+##########################################
+[PR300.CS7.0]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR300.CS7.0_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR300.CS7.0_mapping.txt
+
+##########################################
+[PR300.CS7.1]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR300.CS7.1_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR300.CS7.1_mapping.txt
+
+##########################################
+[PR500.CS5.3]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR500.CS5.3_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR500.CS5.3_mapping.txt
+
+##########################################
+[PR500.CS5.2]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR500.CS5.2_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR500.CS5.2_mapping.txt
+
+##########################################
+[PR500.CS5.1]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR500.CS5.1_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR500.CS5.1_mapping.txt
+
+##########################################
+[PR500.CS5.0]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_PR500.CS5.0_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/PR500.CS5.0_mapping.txt
+
+##########################################
+[KJ100]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_KJ100.CS5_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/KJ100_mapping.txt
+
+##########################################
+[COP22]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_COPV22_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/COPV22_analyte_mapping.txt
+
+##########################################
+[COP23]
+cell_set_definition_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/cell_set_definitions/PRISM_COPV23_definition.txt
+analyte_mapping_file = https://s3.amazonaws.com/analysis.clue.io/vdb/merino/analyte_mapping/COPV23_analyte_mapping.txt
+
+##########################################
+[required_metadata_fields]
+row_metadata_fields = ['ccle_name', 'davepool_id', 'analyte_id', 'barcode_id', 'cell_iname', 'pool_id']
+column_metadata_fields = ['pert_id', 'pert_dose', 'pert_type', 'pert_well', 'pert_dose_unit',
+ 'pert_iname']
+
+##########################################
+[headers_to_pull]
+cell_set_definition_headers = ['analyte_id', 'pool_id', 'davepool_id', 'feature_id', 'cell_iname', 'ccle_name', 'cell_lineage', 'cell_culture', 'barcode_id']
+analyte_mapping_headers = ['analyte_id', 'feature_id']
diff --git a/assemble/parse_data.py b/assemble/parse_data.py
new file mode 100644
index 0000000..b4d27ba
--- /dev/null
+++ b/assemble/parse_data.py
@@ -0,0 +1,92 @@
+
+import urllib3
+import setup_logger as setup_logger
+import logging
+import utils.path_utils as path_utils
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+def parse_data(header_map, data, BuildClass):
+ r = []
+ for row in data:
+ bc = BuildClass()
+ r.append(bc)
+ for (h,i) in header_map.items():
+ if len(row) > i:
+ raw_value = row[i]
+ val = parse_raw_value(raw_value)
+
+ bc.__dict__[h] = val
+
+ return r
+
+def parse_json(data, BuildClass):
+ r = []
+ for obj in data:
+ bc = BuildClass()
+ r.append(bc)
+ for k,v in obj.items():
+ bc.__dict__[k] = parse_raw_value(v)
+
+ #bc.__dict__ = { k:parse_raw_value(v) for k,v in obj.items() }
+ return r
+
+def parse_raw_value(raw_value):
+ val = raw_value
+ if not val:
+ val = None
+ else:
+ try:
+ val = float(val)
+ except ValueError:
+ pass
+
+ return val
+
+#todo: re-write to pull out subset of headers, header mapping no longer necessary
+def generate_header_map(headers, internal_header_file_header_pairs, do_keep_all):
+ """
+ creates a mapping between all headers and a subset of headers (indicated by internal_header_file_header_pairs)
+ allowing for changes to be made to the original header naming scheme (a currently deprecated utility)
+
+ :param headers: (list of strings) list of all headers in file
+ :param internal_header_file_header_pairs: (list of tuples) allows headers to be renamed
+ :param do_keep_all: (boolean) if false uses internal_header_file_header_pairs to pull out subset of headers
+ :return:
+ """
+ reverse_header_map = {}
+ header_map = {}
+
+ if internal_header_file_header_pairs is not None:
+ for (c_key, c_header_name) in internal_header_file_header_pairs:
+ reverse_header_map[c_header_name] = c_key
+
+ for (i, h) in enumerate(headers):
+ if h in reverse_header_map:
+ c_key = reverse_header_map[h]
+ header_map[c_key] = i
+ elif do_keep_all:
+ header_map[h] = i
+
+ if len(header_map) == 0:
+ raise Exception("prism_metadata _generate_header_map header_map has no entries, possible mismatch between expected and actual columns")
+
+ return header_map
+
+
+def read_data(tsv_filepath):
+ tsv_filepath = path_utils.validate_path_as_uri(tsv_filepath)
+
+ f = urllib3.urlopen(tsv_filepath)
+ raw_data = f.read().strip().split("\n")
+ f.close()
+
+ split_raw_data = [x.split("\t") for x in raw_data]
+
+ headers = [x.lower() for x in split_raw_data[0]]
+ logger.debug("headers: {}".format(headers))
+
+ data = split_raw_data[1:]
+ return (headers, data)
+
+
+
diff --git a/assemble/prism_metadata.py b/assemble/prism_metadata.py
new file mode 100644
index 0000000..31f2e7c
--- /dev/null
+++ b/assemble/prism_metadata.py
@@ -0,0 +1,261 @@
+import setup_logger as setup_logger
+import os
+import ast
+import pandas
+import logging
+import requests
+import parse_data
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+API_URL = 'https://api.clue.io/api/'
+API_KEY = os.environ['API_KEY']
+
+CELL_SET_DEFINITION_HEADERS = [
+ 'analyte_id',
+ 'pool_id',
+ 'davepool_id',
+ 'feature_id',
+ 'cell_iname',
+ 'ccle_name',
+ 'cell_lineage',
+ 'cell_culture',
+ 'barcode_id'
+]
+
+class PrismCell(object):
+ def __init__(self, pool_id=None, analyte_id=None, davepool_id=None, feature_id=None):
+ self.pool_id = pool_id
+ self.analyte_id = analyte_id
+ self.davepool_id = davepool_id
+ self.feature_id = feature_id
+
+ self.ignore = False
+
+ def __repr__(self):
+ return " ".join(["{}:{}".format(str(k),str(v)) for (k,v) in self.__dict__.items()])
+
+ def __str__(self):
+ return self.__repr__()
+
+ def validate_properties(self, expected_properties):
+ for property in expected_properties:
+ if hasattr(self,property):
+ continue
+ else:
+ raise Exception("missing property: {}".format(property))
+
+
+class Perturbagen(object):
+ def __init__(self, pert_well=None):
+ self.pert_well = pert_well
+
+ def __repr__(self):
+ return " ".join(["{}:{}".format(str(k),str(v)) for (k,v) in self.__dict__.items()])
+
+ def __str__(self):
+ return self.__repr__()
+
+ def validate_properties(self, expected_properties):
+ for property in expected_properties:
+ if hasattr(self,property):
+ continue
+ else:
+ raise Exception("missing property: {}".format(property))
+
+
+def make_request_url_filter(endpoint_url, where=None, fields=None):
+ clauses = []
+ if where:
+ where_clause = '"where":{'
+ wheres = []
+ for k,v in where.items():
+ wheres.append('"{k}":"{v}"'.format(k=k, v=v))
+ where_clause += ','.join(wheres) + '}'
+ clauses.append(where_clause)
+
+ if fields:
+ fields_clause = '"fields":{'
+ fields_list = []
+ if type(fields) == dict:
+ for k, v in fields.items():
+ fields_list.append('"{k}":"{v}"'.format(k=k, v=v))
+ elif type(fields) == list:
+ for field in fields:
+ fields_list.append('"{k}":"{v}"'.format(k=field, v="true"))
+ fields_clause += ','.join(fields_list) + '}'
+ clauses.append(fields_clause)
+
+ if len(clauses) > 0:
+ #print(endpoint_url.rstrip("/") + '?filter={' + ','.join(clauses) + '}')
+ return endpoint_url.rstrip("/") + '?filter={' + requests.utils.quote(','.join(clauses)) + '}'
+ else:
+ return endpoint_url
+
+
+def get_data_from_db(endpoint_url, user_key, where = None, fields=None):
+ request_url = make_request_url_filter(endpoint_url, where = where, fields=fields)
+ logger.debug(request_url)
+ # print(request_url)
+ response = requests.get(request_url, headers={'user_key': user_key})
+ if response.ok:
+ return response.json()
+ else:
+ response.raise_for_status()
+
+def read_prism_cell_from_file(row_metadata_file, items):
+
+ filepath = row_metadata_file
+
+ (headers, data) = parse_data.read_data(filepath)
+
+ data = [x for x in data if x[0][0] != "#"]
+
+ header_map = parse_data.generate_header_map(headers, items, False)
+
+ logger.debug("header_map: {}".format(header_map))
+ return parse_data.parse_data(header_map, data, PrismCell)
+
+def build_prism_cell_list(config_parser, cell_set_definition_file):
+ '''
+ read PRISM cell line metadata from file specified in config file, then associate with
+ assay_plate based on pool ID, pulling out metadata based on config specifications. Check for cell pools that are not associated with any assay plate
+ :param config_parser: parser pre-loaded with config file
+ :param cell_set_definition_file:
+ :param analyte_mapping_file:
+ :return:
+ '''
+
+ #read headers to pull from config and convert to tuple format expected by data parser
+ prism_cell_list_items = config_parser.get("headers_to_pull", "cell_set_definition_headers")
+
+ prism_cell_list_items = [(x,x) for x in ast.literal_eval(prism_cell_list_items)]
+ prism_cell_list = read_prism_cell_from_file(cell_set_definition_file, prism_cell_list_items)
+
+ return prism_cell_list
+
+def _read_prism_cell_from_db(cell_set_name, beadset, api_url):
+ cell_set_def_url = api_url + 'cell_set_definition_files/'
+ data = get_data_from_db(
+ cell_set_def_url,
+ where={'davepool_id':cell_set_name, 'beadset':beadset},
+ fields= CELL_SET_DEFINITION_HEADERS,
+ user_key=API_KEY)
+
+ return parse_data.parse_json(data, PrismCell)
+
+def build_prism_cell_list_from_db(cell_set_name, beadset, api_url):
+ '''
+ read PRISM cell line metadata from file specified in config file, then associate with
+ assay_plate based on pool ID, pulling out metadata based on config specifications. Check for cell pools that are not associated with any assay plate
+ :param config_parser: parser pre-loaded with config file
+ :param cell_set_definition_file:
+ :param analyte_mapping_file:
+ :return:
+ '''
+
+ #read headers to pull from config and convert to tuple format expected by data parser
+ prism_cell_list = _read_prism_cell_from_db(cell_set_name,beadset, api_url)
+
+ if not prism_cell_list:
+ raise ValueError("cell set name not found in db (davepool_id): {}".format(cell_set_name))
+
+ return prism_cell_list
+
+
+def build_perturbagens_from_file(filepath, pert_time):
+
+ perturbagens = _read_perturbagen_from_file(filepath, do_keep_all=True)
+ _add_pert_time_info(perturbagens, pert_time)
+
+ return perturbagens
+
+def build_perturbagens_from_db(map_src_name, pert_time, api_url):
+
+ perturbagens = _read_perturbagen_from_db(map_src_name, api_url)
+ _add_pert_time_info(perturbagens, pert_time)
+
+ if not perturbagens:
+ raise ValueError("Map Src name not found (pert_plate, replicate): {}".format(map_src_name))
+
+ return perturbagens
+
+
+def _read_perturbagen_from_file(filepath, do_keep_all):
+
+ (headers, data) = parse_data.read_data(filepath)
+
+ #todo: think about other checks / better notification of wrong map type
+ if "well_position" in headers:
+ Exception("Merino no longer supports CM map type, please convert map to CMap map type")
+
+ header_map = parse_data.generate_header_map(headers, None, do_keep_all)
+ logger.debug("header_map: {}".format(header_map))
+
+ return parse_data.parse_data(header_map, data, Perturbagen)
+
+def _read_perturbagen_from_db(map_src_name, api_url):
+ tok = map_src_name.split('.')
+ pert_plate, replicate = '.'.join(tok[:-1]),tok[-1]
+
+ map_src_url = api_url + 'v_plate_map_src/'
+ data = get_data_from_db(
+ map_src_url,
+ user_key=API_KEY,
+ where={'pert_plate': pert_plate, 'replicate': replicate}
+ )
+
+ return parse_data.parse_json(data, Perturbagen)
+
+
+
+def _add_pert_time_info(perturbagens, pert_time):
+
+ pert_time_unit = "h"
+
+ for p in perturbagens:
+ p.pert_time = pert_time
+ p.pert_time_unit = pert_time_unit
+ p.pert_itime = p.pert_time + " " + p.pert_time_unit
+
+
+def convert_objects_to_metadata_df(index_builder, object_list, meta_renaming_map):
+ """
+
+ :param index_builder: Function that given a provided entry in object_list provides a unique index for that entry
+ :param object_list: List of objects that should be converted into rows in the data frame. The properties of these
+ objects will be the columns of the data frame.
+ :param meta_renaming_map: A mapping between the name of a property and the column header to be used in the output.
+ :return: A dataframe where each row corresponds to one of the objects in the object list.
+ """
+ logger.debug("len(object_list): {}".format(len(object_list)))
+
+ col_metadata_map = {}
+ for p in object_list:
+ for k in p.__dict__.keys():
+ if k not in col_metadata_map:
+ col_metadata_map[k] = []
+
+ logger.debug("col_metadata_map.keys(): {}".format(col_metadata_map.keys()))
+ index = []
+ for p in object_list:
+ index.append(index_builder(p))
+
+ for (field, list) in col_metadata_map.items():
+ value = p.__dict__[field] if field in p.__dict__ else None
+ list.append(value)
+
+ if meta_renaming_map is not None:
+ for (original_name, new_name) in meta_renaming_map.items():
+ if new_name not in col_metadata_map:
+ col_metadata_map[new_name] = col_metadata_map[original_name]
+ del col_metadata_map[original_name]
+ else:
+ raise Exception("prism_metadata convert_perturbagen_list_to_col_metadata_df conflict in column names - renaming "
+ "column will erase existing data. col_meta_renaming_map: {} col_metadata_map.keys(): {}".format(
+ meta_renaming_map, col_metadata_map.keys()
+ ))
+
+ return pandas.DataFrame(col_metadata_map, index=index)
+
+
diff --git a/assemble/setup_logger.py b/assemble/setup_logger.py
new file mode 100644
index 0000000..bbd76d0
--- /dev/null
+++ b/assemble/setup_logger.py
@@ -0,0 +1,24 @@
+__author__ = 'dlahr'
+
+import logging
+import logging.handlers
+
+LOGGER_NAME = "prism_logger"
+
+_LOG_FORMAT = "%(levelname)s %(asctime)s %(module)s %(funcName)s %(message)s"
+_LOG_FILE_MAX_BYTES = 10000000
+_LOG_FILE_BACKUP_COUNT = 5
+
+logger = logging.getLogger(LOGGER_NAME)
+
+def setup(verbose=False, log_file=None):
+ level = (logging.DEBUG if verbose else logging.INFO)
+
+ if log_file is None:
+ logging.basicConfig(level=level, format=_LOG_FORMAT)
+ else:
+ logger.setLevel(level)
+ handler = logging.handlers.RotatingFileHandler(log_file, maxBytes=_LOG_FILE_MAX_BYTES,
+ backupCount=_LOG_FILE_BACKUP_COUNT)
+ handler.setFormatter(logging.Formatter(fmt=_LOG_FORMAT))
+ logger.addHandler(handler)
diff --git a/assemble/tests/test_assemble.py b/assemble/tests/test_assemble.py
new file mode 100644
index 0000000..17894dc
--- /dev/null
+++ b/assemble/tests/test_assemble.py
@@ -0,0 +1,160 @@
+import assemble
+import unittest
+import merino.setup_logger as setup_logger
+import logging
+import os
+import glob
+import mock
+
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+
+class TestAssemble(unittest.TestCase):
+
+ def test_full_functional_DP78(self):
+ expected_files = ["assemble/PDOG003_DP78_120H_X1_B1/PDOG003_DP78_120H_X1_B1_COUNT.gct",
+ "assemble/PDOG003_DP78_120H_X1_B1/PDOG003_DP78_120H_X1_B1_MEDIAN.gct",
+ "assemble/PDOG003_DP78_120H_X1_B1/config.yaml",
+ "assemble/PDOG003_DP78_120H_X1_B1/success.txt"]
+ for ef in expected_files:
+ if os.path.exists(ef):
+ os.remove(ef)
+
+
+ config_filepath = "../prism_pipeline.cfg"
+
+ plate_map_path = '../functional_tests/test_data/PDOG/map_src/PDOG003.src'
+ dp7_csv_path = "../functional_tests/test_data/PDOG/lxb/PDOG003_DP7_120H_X1_B1/PDOG003_DP7_120H_X1_B1.csv"
+ dp8_csv_path = "../functional_tests/test_data/PDOG/lxb/PDOG003_DP8_120H_X1_B1/PDOG003_DP8_120H_X1_B1.csv"
+ assay_type = "DP78"
+
+ args = assemble.build_parser().parse_args(["-config_filepath", config_filepath,
+ "-pmp", plate_map_path, "-dp_csv", "DP7", dp7_csv_path, "DP8", dp8_csv_path,
+ "-at", assay_type])
+
+ logger.debug("args: {}".format(args))
+
+ assemble.main(args)
+
+ for ef in expected_files:
+ assert os.path.exists(ef), ef
+ os.remove(ef)
+ os.rmdir("assemble/PDOG003_DP78_120H_X1_B1")
+ os.rmdir("assemble")
+
+ #for map_file in glob.glob('PDOG*.src'):
+ # x = os.path.getsize(map_file)
+ # assert x > 0
+ # os.remove(map_file)
+
+ def test_full_functional_PR500(self):
+ expected_files = ["assemble/PASG003_PR500.2_120H_X251_BX/PASG003_PR500.2_120H_X251_BX_COUNT.gct",
+ "assemble/PASG003_PR500.2_120H_X251_BX/PASG003_PR500.2_120H_X251_BX_MEDIAN.gct",
+ "assemble/PASG003_PR500.2_120H_X251_BX/config.yaml",
+ "assemble/PASG003_PR500.2_120H_X251_BX/success.txt"]
+ for ef in expected_files:
+ if os.path.exists(ef):
+ os.remove(ef)
+
+ config_filepath = "../prism_pipeline.cfg"
+
+ plate_map_path = '../functional_tests/test_data/PASG/map_src/PASG003.src'
+ csv_path = "../functional_tests/test_data/PASG/lxb/PASG003_PR500.2_120H_X251_BX/PASG003_PR500.2_120H_X251_BX.csv"
+ assay_type = "PR500"
+
+ args = assemble.build_parser().parse_args(["-config_filepath", config_filepath, "-pmp", plate_map_path,
+ "-csv", csv_path, "-at", assay_type])
+
+ logger.debug("args: {}".format(args))
+
+ assemble.main(args)
+
+ for ef in expected_files:
+ assert os.path.exists(ef), ef
+ os.remove(ef)
+ os.rmdir("assemble/PASG003_PR500.2_120H_X251_BX")
+ os.rmdir("assemble")
+
+ for map_file in glob.glob('PASG*.src'):
+ x = os.path.getsize(map_file)
+ assert x > 0
+ os.remove(map_file)
+
+
+ def test_full_functional_PR300(self):
+ # the plate_map in test_data is CM map, and thus fails header checks within validate_prism_gct
+ # mocking the column metadata fields within output validation function given
+ # the real test here is analyte and cell set mapping
+
+ expected_files = ["assemble/PSPA001_PR300_120H_X1_BX/PSPA001_PR300_120H_X1_BX_COUNT.gct",
+ "assemble/PSPA001_PR300_120H_X1_BX/PSPA001_PR300_120H_X1_BX_MEDIAN.gct",
+ "assemble/PSPA001_PR300_120H_X1_BX/config.yaml",
+ "assemble/PSPA001_PR300_120H_X1_BX/success.txt"]
+
+ for ef in expected_files:
+ if os.path.exists(ef):
+ os.remove(ef)
+
+ config_filepath = "../prism_pipeline.cfg"
+
+ plate_map_path = '../functional_tests/test_data/PSPA/map_src/PSPA001.src'
+ csv_path = "../functional_tests/test_data/PSPA/lxb/PSPA001_PR300_120H_X1_BX/PSPA001_PR300_120H_X1_BX.csv"
+ assay_type = "PR300"
+
+ args = assemble.build_parser().parse_args(["-config_filepath", config_filepath, "-pmp", plate_map_path,
+ "-csv", csv_path, "-at", assay_type])
+
+ logger.debug("args: {}".format(args))
+
+ assemble.main(args)
+
+ for ef in expected_files:
+ assert os.path.exists(ef), ef
+ os.remove(ef)
+ os.rmdir("assemble/PSPA001_PR300_120H_X1_BX")
+ os.rmdir("assemble")
+
+ for map_file in glob.glob('PSPA*.src'):
+ x = os.path.getsize(map_file)
+ assert x > 0
+ os.remove(map_file)
+
+ def test_full_functional_COPRO(self):
+ expected_files = ["assemble/PGUM001_KJ100_120H_X1_BX/PGUM001_KJ100_120H_X1_BX_COUNT.gct",
+ "assemble/PGUM001_KJ100_120H_X1_BX/PGUM001_KJ100_120H_X1_BX_MEDIAN.gct",
+ "assemble/PGUM001_KJ100_120H_X1_BX/config.yaml",
+ "assemble/PGUM001_KJ100_120H_X1_BX/success.txt"]
+
+ for ef in expected_files:
+ if os.path.exists(ef):
+ os.remove(ef)
+
+ config_filepath = "../prism_pipeline.cfg"
+
+ plate_map_path = '../functional_tests/test_data/PGUM/map_src/PGUM001.src'
+ csv_path = "../functional_tests/test_data/PGUM/lxb/PGUM001_KJ100_120H_X1_BX/PGUM001_KJ100_120H_X1_BX.jcsv"
+ assay_type = "KJ100"
+
+ args = assemble.build_parser().parse_args(["-config_filepath", config_filepath, "-pmp", plate_map_path,
+ "-csv", csv_path, "-at", assay_type])
+
+ logger.debug("args: {}".format(args))
+
+ assemble.main(args)
+
+ for ef in expected_files:
+ assert os.path.exists(ef), ef
+ os.remove(ef)
+ os.rmdir("assemble/PGUM001_KJ100_120H_X1_BX")
+ os.rmdir("assemble")
+
+ for map_file in glob.glob('PGUM*.src'):
+ x = os.path.getsize(map_file)
+ assert x > 0
+ os.remove(map_file)
+
+if __name__ == "__main__":
+ setup_logger.setup(verbose=True)
+
+ unittest.main()
diff --git a/assemble/tests/test_assemble_core.py b/assemble/tests/test_assemble_core.py
new file mode 100644
index 0000000..e5d5640
--- /dev/null
+++ b/assemble/tests/test_assemble_core.py
@@ -0,0 +1,204 @@
+import assemble_core
+import unittest
+import merino.setup_logger as setup_logger
+import logging
+import prism_metadata
+import davepool_data
+import numpy
+import os
+import glob
+
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+
+class TestAssembleWithObjects(unittest.TestCase):
+ def test_build_davepool_to_cells_map(self):
+ l = [prism_metadata.PrismCell(pool_id=str(x + 20), analyte_id=str(x + 10), davepool_id=str(x / 2)) for x in range(6)]
+ logger.debug("l: {}".format(l))
+
+ r = assemble_core.build_davepool_id_to_cells_map(l)
+ assert len(r) == 3, len(r)
+ logger.debug("r: {}".format(r))
+
+ for x in range(3):
+ y = str(x)
+ assert y in r, y
+ assert len(r[y]) == 2, r[y]
+
+ def test_build_data_by_cell(self):
+ cells = [prism_metadata.PrismCell(pool_id=str(x + 20), analyte_id=str(x + 10), davepool_id=str(x / 2)) for x in range(2)]
+ logger.debug("cells: {}".format(cells))
+
+ davepool_data_obj = davepool_data.DavepoolData()
+ davepool_data_obj.median_headers = ["Location", "10", "11"]
+ davepool_data_obj.median_headers.extend([str(x) for x in range(30,40)])
+
+ davepool_data_obj.median_data = {"A01": [1,2], "J13": [3,5]}
+ davepool_data_obj.median_data['A01'].extend([x for x in range(40,50)])
+ davepool_data_obj.median_data['J13'].extend([x for x in range(50,60)])
+
+ davepool_data_obj.count_headers = davepool_data_obj.median_headers
+ davepool_data_obj.count_data = {"B03": [7,11], "L17": [13,17]}
+ davepool_data_obj.count_data["B03"].extend([x for x in range(60,70)])
+ davepool_data_obj.count_data["L17"].extend([x for x in range(70,80)])
+ logger.debug("davepool_data_obj: {}".format(davepool_data_obj))
+
+ rep = assemble_core.build_data_by_cell(cells, davepool_data_obj)
+
+ r_med = rep[0]
+ assert r_med is not None
+ logger.debug("r_med: {}".format(r_med))
+ assert r_med.well_list is not None
+ assert len(r_med.well_list) == 2, len(r_med.well_list)
+ assert "A01" in r_med.well_list, r_med.well_list
+ assert "J13" in r_med.well_list, r_med.well_list
+ assert len(r_med.cell_data_map) == len(cells), (len(r_med.cell_data_map), len(cells))
+
+
+ r_count = rep[1]
+ assert len(r_count.well_list) == 2, len(r_count.well_list)
+ logger.debug("r_count.well_list: {}".format(r_count.well_list))
+ assert "B03" in r_count.well_list, r_count.well_list
+ assert "L17" in r_count.well_list, r_count.well_list
+
+ assert len(r_count.cell_data_map) == len(cells), (len(r_count.cell_data_map), len(cells))
+ logger.debug("r_count.cell_data_map: {}".format(r_count.cell_data_map))
+
+ def test_process_data(self):
+ cells = [prism_metadata.PrismCell(pool_id=str(x + 20), analyte_id=str(x + 10), davepool_id=str(x / 2)) for x in range(4)]
+ logger.debug("cells: {}".format(cells))
+ cells_map = assemble_core.build_davepool_id_to_cells_map(cells)
+ logger.debug("cells_map: {}".format(cells_map))
+
+ davepool_list = []
+ davepool_data_obj = davepool_data.DavepoolData()
+ davepool_list.append(davepool_data_obj)
+
+ davepool_data_obj.davepool_id = "0"
+ davepool_data_obj.median_headers = [str(x) for x in range(10,22)]
+ davepool_data_obj.median_headers.insert(0, "Location")
+ davepool_data_obj.median_data = {"A01": [1,2], "J13": [3,5]}
+ davepool_data_obj.median_data["A01"].extend([x for x in range(40,50)])
+ davepool_data_obj.median_data["J13"].extend([x for x in range(50,60)])
+
+ davepool_data_obj.count_headers = davepool_data_obj.median_headers
+ davepool_data_obj.count_data = {"A01": [7,11], "J13": [13,17]}
+ davepool_data_obj.count_data["A01"].extend([x for x in range(60,70)])
+ davepool_data_obj.count_data["J13"].extend([x for x in range(70,80)])
+ logger.debug("davepool_data_obj: {}".format(davepool_data_obj))
+
+ davepool_data_obj = davepool_data.DavepoolData()
+ davepool_list.append(davepool_data_obj)
+
+ davepool_data_obj.davepool_id = "1"
+ davepool_data_obj.median_headers = [str(x) for x in range(10,22)]
+ davepool_data_obj.median_headers.insert(0, "Location")
+ davepool_data_obj.median_data = {"A01": [-1,-2,19,23], "J13": [-3,-5,29,31]}
+ davepool_data_obj.median_data["A01"].extend([x for x in range(80,88)])
+ davepool_data_obj.median_data["J13"].extend([x for x in range(90,98)])
+
+ davepool_data_obj.count_headers = davepool_data_obj.median_headers
+ davepool_data_obj.count_data = {"A01": [-7,-11,37,41], "J13": [-13,-17,43,47]}
+ davepool_data_obj.count_data["A01"].extend([x for x in range(100,108)])
+ davepool_data_obj.count_data["J13"].extend([x for x in range(110,118)])
+ logger.debug("davepool_data_obj: {}".format(davepool_data_obj))
+
+ r = assemble_core.process_data(davepool_list, cells_map)
+ assert len(r) == 2, len(r)
+
+ median_data_by_cell = r[0]
+ assert median_data_by_cell is not None
+ logger.debug("median_data_by_cell: {}".format(median_data_by_cell))
+ assert 2 == median_data_by_cell.cell_data_map[cells[1]][0]
+ assert 3 == median_data_by_cell.cell_data_map[cells[0]][1]
+ t_wells = median_data_by_cell.well_list
+ logger.debug("t_wells: {}".format(t_wells))
+ assert "J13" in t_wells
+
+ assert 23 == median_data_by_cell.cell_data_map[cells[3]][0]
+ assert 29 == median_data_by_cell.cell_data_map[cells[2]][1]
+
+ def test_build_gctoo(self):
+ filepath = "functional_tests/test_assemble/test_write_output_gct.txt"
+ if os.path.exists(filepath):
+ os.remove(filepath)
+
+ prn = "my_prism_replicate"
+
+ pert_list = []
+ p = prism_metadata.Perturbagen()
+ p.pert_well= "J01"
+ p.pert_id = "BRD-K12345678"
+ p.pert_dose = 11.0
+ p.pert_dose_unit = "uM"
+ pert_list.append(p)
+ p = prism_metadata.Perturbagen()
+ p.pert_well = "M03"
+ p.pert_id = "BRD-K91011121"
+ p.pert_dose = 13.0
+ p.pert_dose_unit = "mg/mL"
+ pert_list.append(p)
+ p = prism_metadata.Perturbagen()
+ p.pert_well = "B02"
+ p.pert_id = "BRD-K31415171"
+ p.pert_dose = 17.0
+ p.pert_dose_unit = "nM"
+ pert_list.append(p)
+
+ cell_list = []
+ c = prism_metadata.PrismCell(pool_id="fake pool", analyte_id="fake analyte",
+ davepool_id="fake davepool", feature_id='c-3')
+ cell_list.append(c)
+ c = prism_metadata.PrismCell(pool_id="fake pool 2", analyte_id="fake analyte 2",
+ davepool_id="fake davepool 2", feature_id='c-5')
+ cell_list.append(c)
+ c = prism_metadata.PrismCell(pool_id="fake pool 2", analyte_id="fake analyte 3",
+ davepool_id="fake davepool 2", feature_id='c-7')
+ cell_list.append(c)
+
+ data_by_cell = assemble_core.DataByCell({cell_list[1]:[1, 2, 11], cell_list[2]:[13, 17, 19], cell_list[0]:[23, 29, 31]},
+ ["J01", "M03", "B02"])
+
+ r = assemble_core.build_gctoo(prn, pert_list, data_by_cell)
+ self.assertIsNotNone(r)
+ logger.debug("r: {}".format(r))
+ logger.debug("r.col_metadata_df: {}".format(r.col_metadata_df))
+ logger.debug("r.row_metadata_df: {}".format(r.row_metadata_df))
+ logger.debug("r.data_df: {}".format(r.data_df))
+
+ def test_build_gctoo_data_df(self):
+ #happy path - all numbers
+ cell_id_data_map = {"cell1":[0,1,2,3], "cell2":[5,7,11,13]}
+ data_df_column_ids = ["s1", "s2", "s3", "s5"]
+ r = assemble_core.build_gctoo_data_df(cell_id_data_map, data_df_column_ids)
+ self.assertIsNotNone(r)
+ logger.debug("r: {}".format(r))
+ self.assertEquals(True, all(data_df_column_ids == r.columns),
+ "columns of result do not match provided columns - data_df_column_ids: {} r.columns: {}".format(
+ data_df_column_ids, r.columns))
+ #check that index entries are present and sorted
+ self.assertEquals("cell1", r.index[0])
+ self.assertEquals("cell2", r.index[1])
+
+ #2nd happy path - blank entry present in data
+ cell_id_data_map["cell1"][1] = ""
+ cell_id_data_map["cell2"][2] = ""
+ r = assemble_core.build_gctoo_data_df(cell_id_data_map, data_df_column_ids)
+ self.assertIsNotNone(r)
+ logger.debug("r: {}".format(r))
+ s = r["s2"].loc["cell1"]
+ logger.debug("s: {}".format(s))
+ self.assertEquals(assemble_core._NaN, s, "expected blank empty string to be replaced with _NaN, was not")
+
+ s = r["s3"].loc["cell2"]
+ logger.debug("s: {}".format(s))
+ self.assertEquals(assemble_core._NaN, s, "expected blank empty string to be replaced with _NaN, was not")
+
+
+
+
+if __name__ == "__main__":
+ setup_logger.setup(verbose=True)
+
+ unittest.main()
\ No newline at end of file
diff --git a/assemble/tests/test_davepool_data.py b/assemble/tests/test_davepool_data.py
new file mode 100644
index 0000000..7c7fa12
--- /dev/null
+++ b/assemble/tests/test_davepool_data.py
@@ -0,0 +1,96 @@
+import logging
+import merino.setup_logger as setup_logger
+import unittest
+import davepool_data
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+
+class TestDavepoolData(unittest.TestCase):
+ def test___init__(self):
+ r = davepool_data.DavepoolData()
+ assert hasattr(r, "median_headers")
+
+ def test_read_data(self):
+ csv_filepath = "../functional_tests/test_data/PASG/lxb/PASG003_PR500.2_120H_X251_BX/PASG003_PR500.2_120H_X251_BX.csv"
+ r = davepool_data.read_data(csv_filepath)
+ assert r is not None
+ assert r.csv_filepath == csv_filepath, r.csv_filepath
+ assert r.csv_datetime is not None
+ assert r.median_headers is not None
+ assert r.median_data is not None
+ assert r.count_headers is not None
+ assert r.count_data is not None
+
+ logger.debug("r.csv_datetime: {}".format(r.csv_datetime))
+
+ def test_read_data_from_jcsv(self):
+ csv_filepath = "../functional_tests/test_data/PGUM/lxb/PGUM001_KJ100_120H_X1_BX/PGUM001_KJ100_120H_X1_BX.jcsv"
+ r = davepool_data.read_data(csv_filepath)
+
+ assert len(r.count_data) == 384, len(r.count_data)
+
+ def test_get_datatype_range(self):
+ data = [["a"], range(3), ["DataType:","my type"], range(4), range(5)]
+
+ r = davepool_data.get_datatype_range(data, ["my type"])
+
+ assert r is not None
+ logger.debug("r: {}".format(r))
+ assert "my type" in r
+ r = r["my type"]
+ assert r[0] == 2, r[0]
+ assert r[1] == 6, r[1]
+
+ r = davepool_data.get_datatype_range(data, ["unfounded"])
+ assert r is not None
+ logger.debug("r: {}".format(r))
+ assert "unfounded" in r
+ r = r["unfounded"]
+ assert r[0] is None, r[0]
+ assert r[1] is None, r[1]
+
+ data.append(["DataType:", "another type"])
+ data.append(range(6))
+ r = davepool_data.get_datatype_range(data, ["my type"])
+
+ assert r is not None
+ logger.debug("r: {}".format(r))
+ assert "my type" in r
+ r = r["my type"]
+ assert r[0] == 2, r[0]
+ assert r[1] == 5, r[1]
+
+ def test_get_datetime_from_header_rows(self):
+ header_rows = [[],["a","b","c"], range(5), [davepool_data.date_header, "first", "second"], [], range(7)]
+ r = davepool_data.get_datetime_from_header_rows(header_rows, "fake csv_filepath")
+ logger.debug("r: {}".format(r))
+ assert r == "first second", r
+
+ header_rows.append([davepool_data.date_header, "problem", "entry"])
+ with self.assertRaises(Exception) as context:
+ davepool_data.get_datetime_from_header_rows(header_rows, "fake csv_filepath")
+ logger.debug("context.exception: {}".format(context.exception))
+ assert "expected to find only one datetime_row, found datetime_row:" in str(context.exception), str(context.exception)
+
+ def test_validate_data(self):
+ dd = davepool_data.DavepoolData()
+
+ #happy path
+ dd.median_data = [[0], [1]]
+ dd.count_data = [[2], [3]]
+ dd.validate_data()
+
+ #invalid data path
+ dd.median_data[1].pop(0)
+ with self.assertRaises(Exception) as context:
+ dd.validate_data()
+ assert context.exception is not None
+ logger.debug("context.exception: {}".format(context.exception))
+ assert "data row is empty" in str(context.exception)
+
+
+if __name__ == "__main__":
+ setup_logger.setup(verbose=True)
+
+ unittest.main()
diff --git a/assemble/tests/test_parse_data.py b/assemble/tests/test_parse_data.py
new file mode 100644
index 0000000..e94d136
--- /dev/null
+++ b/assemble/tests/test_parse_data.py
@@ -0,0 +1,114 @@
+import merino.setup_logger as setup_logger
+import logging
+import unittest
+import configparser
+import parse_data as pd
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+test_file = "functional_tests/test_prism_metadata/prism_cell_tsv.txt"
+
+_prism_cell_config_file_section = "PrismCell column headers"
+_perturbagen_CM_input_config_file_section = "Perturbagen CM input column headers"
+
+class Dummy(object):
+ def __repr__(self):
+ return " ".join(["{}:{}".format(str(k),str(v)) for (k,v) in self.__dict__.items()])
+
+ def __str__(self):
+ return self.__repr__()
+
+
+class TestParseData(unittest.TestCase):
+ def test_read_data(self):
+ (h, d) = pd.read_data(test_file)
+
+ assert h is not None
+ logger.debug("h: {}".format(h))
+ assert "pool_id" in h
+
+ assert d is not None
+ logger.debug("d: {}".format(d))
+ assert len(d) > 0
+
+ def test_generate_header_map(self):
+ #happy path ignore extra field
+ headers = ["pool_id", "analyte", "strippedname", "extra_header"]
+
+ cp = configparser.Rawconfigparser()
+ cp.read("prism_pipeline.cfg")
+ internal_header_file_header_pairs = cp.items(_prism_cell_config_file_section)
+
+ r = pd.generate_header_map(headers, internal_header_file_header_pairs, False)
+ logger.debug("r: {}".format(r))
+ assert len(r) == 3, len(r)
+ assert "extra_header" not in r, r
+ assert "pool_id" in r, r
+ assert r["pool_id"] == 0, r["pool_id"]
+
+ #happy path include extra field
+ r = pd.generate_header_map(headers, internal_header_file_header_pairs, True)
+ logger.debug("r: {}".format(r))
+ assert len(r) == 4, len(r)
+ assert "extra_header" in r
+ assert r["extra_header"] == 3, r["extra_header"]
+
+ def test__parse_data(self):
+ headers = ["pool_id", "analyte", "strippedname"]
+ cp = configparser.Rawconfigparser()
+ cp.read("prism_pipeline.cfg")
+
+ header_map = pd.generate_header_map(headers, cp.items(_prism_cell_config_file_section), False)
+
+ data = [["1", "analyte 2", "my cell's name"], ["3", "analyte 5", "autre cell nom"]]
+
+ r = pd.parse_data(header_map, data, Dummy)
+ logger.debug("r: {}".format(r))
+ assert len(r) == len(data), len(r)
+
+ header_map["extra header that doesn't have data in any row"] = 10
+ r = pd.parse_data(header_map, data, Dummy)
+ logger.debug("r: {}".format(r))
+ assert len(r) == len(data), len(r)
+
+ data.append(["7", "", "blah"])
+ r = pd.parse_data(header_map, data, Dummy)
+ assert r[2].analyte_id is None
+
+ headers = ["well_position", "compound_well_mmoles_per_liter", "dilution_factor"]
+ cp = configparser.Rawconfigparser()
+ cp.read("prism_pipeline.cfg")
+ header_map = pd.generate_header_map(headers, cp.items(_perturbagen_CM_input_config_file_section), False)
+
+ data = [["A01", "1.010101", "2"], ["B07", "3.030303", "5"]]
+ r = pd.parse_data(header_map, data, Dummy)
+ logger.debug("r: {}".format(r))
+ assert len(r) == len(data), len(r)
+
+ assert hasattr(r[0], "compound_well_mmoles_per_liter"), r[0].__dict__
+ assert isinstance(r[0].compound_well_mmoles_per_liter, float)
+
+ assert r[0].compound_well_mmoles_per_liter == 1.010101, r[0].compound_well_mmoles_per_liter
+ assert isinstance(r[0].dilution_factor, int)
+ assert r[0].dilution_factor == 2, r[0].dilution_factor
+ assert isinstance(r[1].compound_well_mmoles_per_liter, float)
+ assert isinstance(r[1].dilution_factor, int)
+
+ def test__parse_raw_value(self):
+ r = pd.parse_raw_value("")
+ assert r is None
+
+ r = pd.parse_raw_value("6")
+ assert r == 6, r
+
+ r = pd.parse_raw_value("6.1")
+ assert r == 6.1, r
+
+ r = pd.parse_raw_value("hello world")
+ assert r == "hello world", r
+
+
+if __name__ == "__main__":
+ setup_logger.setup(verbose=True)
+
+ unittest.main()
diff --git a/assemble/tests/test_prism_metadata.py b/assemble/tests/test_prism_metadata.py
new file mode 100644
index 0000000..8ce7455
--- /dev/null
+++ b/assemble/tests/test_prism_metadata.py
@@ -0,0 +1,233 @@
+import merino.setup_logger as setup_logger
+import logging
+import unittest
+import prism_metadata as pm
+import configparser
+
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+test_file = "functional_tests/test_prism_metadata/prism_cell_tsv.txt"
+
+
+class TestPrismMetadata(unittest.TestCase):
+ def test___init__(self):
+ r = pm.PrismCell()
+ assert hasattr(r, "pool_id")
+ assert hasattr(r, "analyte_id")
+
+ def test_read_prism_cell_from_file(self):
+ cp = configparser.Rawconfigparser()
+ config_filepath = 'prism_pipeline.cfg'
+ cp.read(config_filepath)
+
+ prism_cell_list_items = cp.items("PrismCell column headers")
+ davepool_mapping_items = cp.items("DavepoolAnalyteMapping column headers")
+
+ cell_set_definition = "requirements_artifacts/Tranche1PrimaryMetaData_02252016.txt"
+ r = pm.read_prism_cell_from_file(cell_set_definition, prism_cell_list_items)
+ self.assertGreater(len(r), 20)
+ logger.debug("r[0:10]: {}".format(r[0:10]))
+
+ r = pm.read_prism_cell_from_file("functional_tests/test_prism_metadata/test_read_prism_cell_from_file/test_cell_definition.txt",
+ prism_cell_list_items)
+ self.assertEqual(8, len(r))
+ logger.debug("r[0:8]: {}".format(r[0:10]))
+
+ #check that commented out line is ignored when reading in cell metadata
+ c = [x for x in r if x.id == 4]
+ logger.debug("c: {}".format(c))
+ self.assertEqual(0, len(c), "expected that cell line with id == 5 was not read in because it is commented out in " +
+ "test_cell_definition.txt, but it was found")
+
+ def test__read_perturbagen_from_file_CM(self):
+ r = pm._read_perturbagen_from_file("functional_tests/test_prism_metadata/perturbagen.txt",
+ pm._perturbagen_CM_input_config_file_section, False)
+
+ assert len(r) > 0
+ for x in r:
+ logger.debug("x: {}".format(x))
+
+ def test__read_perturbagen_from_file_CMap(self):
+ #1 perturbagen per well
+ r = pm._read_perturbagen_from_file("functional_tests/test_prism_metadata/LJP005.src",
+ pm._perturbagen_CMap_input_config_file_section, True)
+ assert len(r) > 0, len(r)
+ logger.debug("r[0]: {}".format(r[0]))
+ logger.debug("r[6]: {}".format(r[6]))
+
+ assert r[0].well_id is not None
+ assert hasattr(r[0], "pert_id")
+ assert hasattr(r[0], "pert_type")
+
+ #multiple perturbagens per well
+ r = pm._read_perturbagen_from_file("functional_tests/test_prism_metadata/PMEL.A001.src",
+ pm._perturbagen_CMap_input_config_file_section, True)
+
+ assert len(r) > 0, len(r)
+ logger.debug("r[0]: {}".format(r[0]))
+ logger.debug("r[6]: {}".format(r[6]))
+
+ assert r[0].well_id is not None
+ assert hasattr(r[0], "pert_id")
+ assert hasattr(r[0], "pert_type")
+
+ def test_read_assay_plate_from_file(self):
+ r = pm.read_assay_plate_from_file("functional_tests/test_prism_metadata/assay_plate.txt")
+ assert len(r) > 0
+ for x in r:
+ logger.debug("x: {}".format(x))
+ assert x.assay_plate_barcode is not None
+ assert x.det_plate is not None
+ assert x.pool_id is not None
+
+ def test__build_additional_perturbagen_info(self):
+ p = pm.Perturbagen()
+ #worst case scenario, these are both ints
+ p.compound_well_mmoles_per_liter = int(5)
+ p.dilution_factor = int(2001)
+ p.pert_mfc_id = "BRD-K12345678-910-11-1"
+ p.pert_type = "Test"
+ p.pert_mfc_desc = "my fake compound name"
+
+ pm._build_additional_perturbagen_info([p])
+ logger.debug("p: {}".format(p))
+ assert p.pert_dose == (1000.0 * float(5) / 2001.0), p.pert_dose
+ assert p.pert_dose_unit == "uM", p.pert_dose_unit
+ assert p.pert_idose is not None
+
+ assert p.pert_id == "BRD-K12345678", p.pert_id
+ assert p.pert_type == "trt_cp", p.pert_type
+ assert p.pert_iname == "my fake compound name", p.pert_iname
+ assert p.pert_time == "120", p.pert_time
+ assert p.pert_time_unit == "h", p.pert_time_unit
+ assert p.pert_itime == "120 h", p.pert_itime
+
+ p.pert_mfc_desc = None
+ pm._build_additional_perturbagen_info([p])
+ assert p.pert_iname == p.pert_id, p.pert_iname
+
+ p.pert_type = "unrecognized"
+ pm._build_additional_perturbagen_info([p])
+ assert p.pert_type == "ctl_vehicle", p.pert_type
+
+ p.pert_type = None
+ pm._build_additional_perturbagen_info([p])
+ assert p.pert_type == "ctl_vehicle", p.pert_type
+
+ p.pert_mfc_id = None
+ pm._build_additional_perturbagen_info([p])
+ assert p.pert_id == "DMSO", p.pert_id
+
+ p.compound_well_mmoles_per_liter = "not a valid concentration"
+ with self.assertRaises(Exception) as context:
+ pm._build_additional_perturbagen_info([p])
+ assert context.exception is not None
+ logger.debug("context.exception: {}".format(context.exception))
+ assert "the concentration or dilution factors should be numbers" in str(context.exception), \
+ str(context.exception)
+
+ p.compound_well_mmoles_per_liter = 5.0
+ del p.pert_type
+ with self.assertRaises(Exception) as context:
+ pm._build_additional_perturbagen_info([p])
+ assert context.exception is not None
+ logger.debug("context.exception: {}".format(context.exception))
+ assert "pert_type attribute is missing from perturbagen" in str(context.exception), str(context.exception)
+
+ def test__build_additional_perturbagen_info_vehicle(self):
+ p = pm.Perturbagen()
+ p.pert_type = "Empty"
+ p.pert_vehicle = "DMSO"
+ p.compound_well_mmoles_per_liter = None
+ p.pert_mfc_id = None
+
+ pm._build_additional_perturbagen_info([p])
+ logger.debug("p: {}".format(p))
+
+ assert p.pert_dose is None, p.pert_dose
+ assert p.pert_dose_unit is None, p.pert_dose_unit
+ assert p.pert_idose is None, p.pert_idose
+
+ def test_validate_perturbagens(self):
+ perts = []
+ num_wells = 3
+
+ for assay_plate_barcode in ["a" + str(i) for i in range(2)]:
+ new_perts = [pm.Perturbagen(well_id=i) for i in range(num_wells)]
+ for (i, np) in enumerate(new_perts):
+ np.assay_plate_barcode = assay_plate_barcode
+ np.pert_id = "BRD-K" + str(int(11.0*i))
+ np.pert_idose = str(13.0*i) + " uM"
+ perts.extend(new_perts)
+ logger.debug("perts: {}".format(perts))
+
+ r = pm.validate_perturbagens(perts)
+ logger.debug("r: {}".format(r))
+
+ assert len(r) == num_wells, (len(r), num_wells)
+ for i in range(num_wells):
+ assert i in r
+ assert r[i].well_id == i, r[i]
+
+ perts[num_wells].pert_id = perts[0].pert_id + " extra junk"
+ with self.assertRaises(Exception) as context:
+ pm.validate_perturbagens(perts)
+ assert context.exception is not None
+ logger.debug("context.exception: {}".format(context.exception))
+ assert "the perturbagens provided contain different compounds in the same wells of different assasy plates" in \
+ str(context.exception), str(context.exception)
+
+ def test_convert_objects_to_metadata_df(self):
+ col_base_id = "my col base id"
+ pert_list = [pm.Perturbagen("A01"), pm.Perturbagen("B02"), pm.Perturbagen(3)]
+ for (i, p) in enumerate(pert_list):
+ p.pert_type = "trt_cp"
+ p.random_inclusion = i
+ p.other = None
+ p.something = i*1.1
+
+ def index_builder(pert):
+ return col_base_id + ":" + str(pert.well_id)
+
+ r = pm.convert_objects_to_metadata_df(index_builder, pert_list)
+ assert r is not None
+ logger.debug("r: {}".format(r))
+
+ assert pm.cmap_pert_well in r.columns
+ assert "pert_type" in r.columns
+ assert "random_inclusion" in r.columns
+ assert "other" in r.columns
+ assert "something" in r.columns
+
+ expected_col_id = index_builder(pert_list[0])
+ assert expected_col_id in r.index, (expected_col_id, r.index)
+ expected_col_id = index_builder(pert_list[2])
+ assert expected_col_id in r.index, (expected_col_id, r.index)
+
+ assert "trt_cp" == r["pert_type"][expected_col_id], r["pert_type"][expected_col_id]
+ assert 2 == r["random_inclusion"][expected_col_id], r["random_inclusion"][expected_col_id]
+ assert r["other"][expected_col_id] is None, r["other"][expected_col_id]
+ assert 2.2 == r["something"][expected_col_id], r["something"][expected_col_id]
+ assert 3 == r["pert_well"][expected_col_id], r["pert_well"][expected_col_id]
+
+ def test_convert_objects_to_metadata_df_for_prism_cell(self):
+ cell_list = [pm.PrismCell(1, 2, 3, 5), pm.PrismCell(7, 11, 13, 17)]
+
+ def index_builder(c):
+ return c.id
+
+ r = pm.convert_objects_to_metadata_df(index_builder, cell_list)
+
+ assert r is not None
+ logger.debug("r: {}".format(r))
+
+ assert "pool_id" in r.columns
+ assert "davepool_id" in r.columns
+ assert "analyte_id" in r.columns
+
+
+if __name__ == "__main__":
+ setup_logger.setup(verbose=True)
+
+ unittest.main()
diff --git a/assemble/utils/__init__.py b/assemble/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/assemble/utils/exceptions.py b/assemble/utils/exceptions.py
new file mode 100644
index 0000000..e685769
--- /dev/null
+++ b/assemble/utils/exceptions.py
@@ -0,0 +1,11 @@
+class DataMappingMismatch(Exception):
+ pass
+
+class UnableToDifferentiateReplicates(Exception):
+ pass
+
+class ReplicateSetSearchFailure(Exception):
+ pass
+
+class NoAssayTypeFound(Exception):
+ pass
diff --git a/assemble/utils/path_utils.py b/assemble/utils/path_utils.py
new file mode 100644
index 0000000..efbb3a1
--- /dev/null
+++ b/assemble/utils/path_utils.py
@@ -0,0 +1,15 @@
+from pathlib import Path
+
+def validate_path_as_uri(path):
+ #ensures that path formatting is amenable to urlopen-ing, i.e. converts to absolute URI
+
+ p = Path(path)
+
+ if p.parts[0] == "https:":
+ return path
+ else:
+ if p.is_absolute():
+ return p.as_uri()
+ else:
+ p = p.resolve()
+ return p.as_uri()
diff --git a/batch_correction/.dockerignore b/batch_correction/.dockerignore
new file mode 100644
index 0000000..d4c43aa
--- /dev/null
+++ b/batch_correction/.dockerignore
@@ -0,0 +1,5 @@
+README.md
+run_batch_correction.sh
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/batch_correction/Dockerfile b/batch_correction/Dockerfile
new file mode 100644
index 0000000..b91a3cc
--- /dev/null
+++ b/batch_correction/Dockerfile
@@ -0,0 +1,15 @@
+FROM cmap/base-clue-mts
+MAINTAINER Andrew Boghossian
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./src/batch_correct_functions.R /src/batch_correct_functions.R
+COPY ./batch_correct.R /batch_correct.R
+COPY ./aws_batch.sh /clue/bin/aws_batch
+
+WORKDIR /
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/batch_correction/README.md b/batch_correction/README.md
new file mode 100644
index 0000000..83c289f
--- /dev/null
+++ b/batch_correction/README.md
@@ -0,0 +1,56 @@
+# Batch Correction Module
+
+This module uses [ComBat](https://www.bioconductor.org/packages/devel/bioc/vignettes/sva/inst/doc/sva.pdf) to correct log-fold change values for batch effects. It requires the output of the LFC module.
+
+## Batch correction overview
+
+Log-fold changes are corrected for batch effects coming from pools and culture conditions using the ComBat algorithm as described in [Johnson et al.](https://pubmed.ncbi.nlm.nih.gov/16632515/). This module is typically run after concatenating PR500 and PR300+ data together.
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/cmap/batch-correct-module) run:
+
+```
+docker pull cmap/batch-correct-module:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `aws_batch.sh` which is a wrapper around `batch_correct.R` for more information using R run
+
+```
+Rscript batch_correct.R --help
+usage: batch_correct.R [-h] [-b BASE_DIR] [-o OUT] [-n NAME]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -b BASE_DIR, --base_dir BASE_DIR
+ Input directory
+ -o OUT, --out OUT Output path. Default is working directory
+ -n NAME, --name NAME Build name. Default is none
+```
+
+### Example usage with R
+
+R execution requires installing the correct R packages which are outlined in the `docker_base` module.
+
+```
+Rscript batch_correct.R -b ~/Desktop/clue_data -o ~/Desktop/mts_results -n PMTS001
+```
+
+### Example usage with Docker
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+```
+docker run \
+ -it \
+ -v ~/Desktop/clue_data:/in_data \
+ -v ~/Desktop/mts_results:/out_data \
+ cmap/batch-correct-module:latest \
+ -b /in_data \
+ -o /out_data \
+ -n PMTS001
+```
diff --git a/batch_correction/aws_batch.sh b/batch_correction/aws_batch.sh
new file mode 100644
index 0000000..fd23cd5
--- /dev/null
+++ b/batch_correction/aws_batch.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+# read in flagged arguments
+while getopts ":b:o:n:" arg; do
+ case $arg in
+ b) # specify input folder
+ data_dir=${OPTARG};;
+ o) # specifcy output folder
+ output_dir=${OPTARG};;
+ n) # specify build name
+ build_name=${OPTARG}
+ esac
+done
+
+chmod +x /batch_correct.R
+chmod +x /src/batch_correct_functions.R
+export HDF5_USE_FILE_LOCKING=FALSE
+echo "${data_dir}" "${output_dir}" "${build_name}"
+Rscript /batch_correct.R -b "${data_dir}" -o "${output_dir}" -n "${build_name}"
+
+exit_code=$?
+
+echo "$exit_code"
+exit $exit_code
diff --git a/batch_correction/batch_correct.R b/batch_correction/batch_correct.R
new file mode 100644
index 0000000..9751494
--- /dev/null
+++ b/batch_correction/batch_correct.R
@@ -0,0 +1,64 @@
+# Script to go from normalized logMFI values to LFC
+
+# import necessary libraries and functions using MTS_functions.R
+suppressMessages(source("./src/batch_correct_functions.R"))
+
+#---- Read arguments ----
+# initialize parser
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-b", "--base_dir", default="", help="Input directory")
+parser$add_argument("-o", "--out", default=getwd(), help = "Output path. Default is working directory")
+parser$add_argument("-n", "--name", default="", help = "Build name. Default is none")
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+
+base_dir <- args$base_dir
+out_dir <- args$out
+build_name <- args$name
+
+if (!dir.exists(out_dir)) {dir.create(out_dir, recursive = T)}
+
+#---- Load the data ----
+print("Loading the data")
+LFC_files <- list.files(base_dir, pattern = "LEVEL4_LFC", full.names = T)
+if (length(LFC_files) == 1) {
+ LFC_TABLE <- data.table::fread(LFC_files[[1]])
+} else {
+ stop(paste("There are", length(LFC_files), "level 4 tables in this directory. Please ensure there is one and try again."),
+ call. = FALSE)
+}
+
+#---- Correct for pool effects ----
+print("ComBat correcting")
+LFC_TABLE %<>%
+ dplyr::filter(!pert_type %in% c("ctl_vehicle", "ctl_untrt"), is.finite(LFC)) %>%
+ tidyr::unite(col = "condition", pert_iname, pert_dose, pert_plate, pert_time, pert_vehicle, any_of("x_project_id"),
+ sep = "::", remove = FALSE) %>%
+ split(.$condition) %>%
+ purrr::map_dfr(~dplyr::mutate(.x, LFC_cb = apply_combat(.))) %>%
+ dplyr::select(-condition)
+
+#---- Make collapsed LFC table ----
+LFC_COLLAPSED_TABLE <- LFC_TABLE %>%
+ dplyr::mutate(sig_id = paste(pert_plate, culture, pert_id, pert_idose, pert_time, sep = fixed("_"))) %>%
+ dplyr::select(rid, ccle_name, culture, pool_id, pert_iname, pert_id, pert_dose,
+ pert_idose, pert_plate, pert_vehicle, pert_time, pert_type, LFC, LFC_cb, sig_id,
+ any_of(c("x_mixture_contents", "x_mixture_id", "x_project_id"))) %>%
+ dplyr::group_by(across(.cols = !contains("LFC"))) %>%
+ # LFC and LFC_cb values will be medians across replicates
+ dplyr::summarize(LFC = median(LFC, na.rm = TRUE),
+ LFC_cb = median(LFC_cb, na.rm = TRUE),
+ .groups = "drop")
+
+dims_full = paste(dplyr::distinct(LFC_TABLE, profile_id) %>% nrow(),
+ dplyr::distinct(LFC_TABLE, rid) %>% nrow(),
+ sep = "x")
+dims_coll = paste(dplyr::distinct(LFC_COLLAPSED_TABLE, sig_id) %>% nrow(),
+ dplyr::distinct(LFC_COLLAPSED_TABLE, rid) %>% nrow(),
+ sep = "x")
+
+#---- Write results ----
+write.csv(LFC_TABLE, paste0(out_dir, "/", build_name, "_LEVEL4_LFC_COMBAT_n", dims_full, ".csv"), row.names=FALSE)
+write.csv(LFC_COLLAPSED_TABLE, paste0(out_dir, "/", build_name, "_LEVEL5_LFC_COMBAT_n", dims_coll, ".csv"), row.names=FALSE)
diff --git a/batch_correction/build_docker.sh b/batch_correction/build_docker.sh
new file mode 100755
index 0000000..dc49f4a
--- /dev/null
+++ b/batch_correction/build_docker.sh
@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+docker build --platform linux/amd64 -t prismcmap/batch-correct-module:latest -t prismcmap/batch-correct-module:v0.0.1 --rm=true .
diff --git a/batch_correction/docker_push_tags.sh b/batch_correction/docker_push_tags.sh
new file mode 100755
index 0000000..b0e53b3
--- /dev/null
+++ b/batch_correction/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/batch-correct-module:v0.0.1
+docker push prismcmap/batch-correct-module:latest
+
+# trigger
\ No newline at end of file
diff --git a/batch_correction/docker_run.sh b/batch_correction/docker_run.sh
new file mode 100755
index 0000000..6a43ba1
--- /dev/null
+++ b/batch_correction/docker_run.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+docker run --rm \
+--name bar2 \
+-v /Users/jasiedu/WebstormProjects/TEST_CPS017_P1000_NEWFMT3/:/cmap/macchiato \
+-v /Users/jasiedu/WebstormProjects/biomarker/.cache/:/cmap/biomarker_cache \
+-e projects="$(cat /Users/jasiedu/WebstormProjects/macchiato/foo1.json)" \
+-e AWS_BATCH_JOB_ARRAY_INDEX=0 \
+-it cmap/batch-correct-module:dev \
+-b /cmap/macchiato/projects \
+-o /cmap/macchiato/projects \
+-d "/cmap/biomarker_cache"
+
+
+
diff --git a/batch_correction/run_batch_correction.sh b/batch_correction/run_batch_correction.sh
new file mode 100755
index 0000000..e3b5f8d
--- /dev/null
+++ b/batch_correction/run_batch_correction.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+docker run -v /Users/jasiedu/WebstormProjects/dockerized_mts/TEST_ALL/:/data/ \
+-it cmap/batch-correct-module:dev \
+-b /data/build \
+-o /data/build \
+-n "TEST_PMTS017_PR1000"
\ No newline at end of file
diff --git a/batch_correction/src/batch_correct_functions.R b/batch_correction/src/batch_correct_functions.R
new file mode 100644
index 0000000..38d79a7
--- /dev/null
+++ b/batch_correction/src/batch_correct_functions.R
@@ -0,0 +1,46 @@
+# Helper functions for the initial processing step of the MTS pipeline
+
+#---- Load Packages ----
+library(tidyverse)
+library(magrittr)
+library(data.table)
+library(sva)
+library(readr)
+library(stats)
+library(reshape2)
+library(argparse)
+
+#---- Batch Correction ----
+# corrects for pool effects using ComBat
+apply_combat <- function(Y) {
+
+ # create "condition" column to be used as "batches"
+ df <- Y %>%
+ dplyr::distinct(ccle_name, prism_replicate, LFC, culture, pool_id, pert_well) %>%
+ tidyr::unite(cond, culture, pool_id, prism_replicate, sep = "::") %>%
+ dplyr::filter(is.finite(LFC))
+
+ # calculate means and sd's of each condition
+ batch <- df$cond
+ m <- rbind(df$LFC,
+ rnorm(length(df$LFC),
+ mean = mean(df$LFC, na.rm = TRUE),
+ sd = sd(df$LFC, na.rm = TRUE)))
+
+ # use ComBat to align means and sd's of conditions
+ combat <- sva::ComBat(dat = m, batch = batch) %>%
+ t() %>%
+ as.data.frame() %>%
+ dplyr::mutate(ccle_name = df$ccle_name, cond = df$cond, pert_well = df$pert_well) %>%
+ dplyr::rename(LFC_cb = V1) %>%
+ dplyr::mutate(culture = stringr::word(cond, 1, sep = stringr::fixed("::")),
+ pool_id = stringr::word(cond, 2, sep = stringr::fixed("::")),
+ prism_replicate = stringr::word(cond, 3, sep = stringr::fixed("::"))) %>%
+ dplyr::select(-cond, -V2)
+
+ combat_corrected <- Y %>%
+ dplyr::left_join(combat, by = c("prism_replicate", "ccle_name", "pool_id", "culture", "pert_well")) %>%
+ .$LFC_cb
+
+ return(combat_corrected)
+}
diff --git a/biomarker/.dockerignore b/biomarker/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/biomarker/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/biomarker/Dockerfile b/biomarker/Dockerfile
new file mode 100644
index 0000000..e6f11ec
--- /dev/null
+++ b/biomarker/Dockerfile
@@ -0,0 +1,15 @@
+FROM cmap/base-clue-mts
+MAINTAINER Andrew Boghossian
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./src/biomarker_functions.R /src/biomarker_functions.R
+COPY ./biomarkers.R /biomarkers.R
+COPY ./aws_batch.sh /clue/bin/aws_batch
+
+WORKDIR /
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/biomarker/README.md b/biomarker/README.md
new file mode 100644
index 0000000..86e284e
--- /dev/null
+++ b/biomarker/README.md
@@ -0,0 +1,62 @@
+
+# Biomarker Module
+
+### Goal
+The goal of this README is to provide an overview on how to run the biomarker analysis locally. The biomarker analysis consists of a univariate analysis and multivariate analysis for the genomic features used in DepMap.
+
+
+### Files
+
+The file used for biomarker analysis is [biomarker.R](https://github.com/cmap/dockerized_mts/blob/master/biomarker/biomarkers.R) found in our [Github](https://github.com/cmap/dockerized_mts/tree/master). The biomarker analysis calls statistical functions found in [cdsr_models](https://github.com/broadinstitute/cdsr_models/tree/master/R).
+
+### Input Data
+To run the the biomarker.R function are the following
+
+1. Log Fold Change (LFC) file
+2. Dose Response Curve (DRC) file
+3. Quality Control (QC) file
+
+These files can be found in your project landing page under Reports and Downloads under the Compound Data tab. Specifically:
+
+1. LFC file: “Replicate-collapsed viability table”
+2. DRC file: “Dose response curve data”
+3. QC file: “QC metrics”
+
+
+
+
+
+
+
+Once these files are downloaded, point them to the ‘lfc_path’, ‘drc_path’, and ‘qc_path’ within the biomarker.R file.
+
+### Running the biomarker.R file:
+
+Now you may run the biomarker.R file. The important variable to keep in mind is the ‘all_Y’ variable, which contains the cell line viabilities by compound and dose. Any customization or subsetting of cell lines should be done using this variable. The three biomarker functions are as follows:
+
+1. **Linear Associations**: computes the correlation coefficient and significance values between continuous genomic features for cancer cell lines and their respective viability profiles. The continuous genomic features are as follows:
+ 1. Gene Expression (GE)
+ 2. CRISPR (XPR)
+ 3. Copy Number (CNA)
+ 4. Metabolomics (MET)
+ 5. Micro RNA (miRNA)
+ 6. Repurposing Library (REP)
+ 7. Proteomics (PROT)
+ 8. Small hairpin RNA (shRNA)
+
+
+A pairwise correlation analysis is conducted through cdsr_models [linear_associations.R](https://github.com/broadinstitute/cdsr_models/blob/master/R/linear_association.R) which the biomarker.R function calls.
+
+2. **Discrete Associations**: computes the effect size and significance values between discrete genomic features for cancer cell lines and their respective viability profiles. the discrete genomic features are as follows:
+ 1. Lineage (LIN)
+ 2. Mutation (MUT)
+
+For these categorical features, a t-test is conducted between a specific LIN/MUT and all others to obtain effect size and significance through cdsr_models [discrete_associations.R](https://github.com/broadinstitute/cdsr_models/blob/master/R/discrete_association.R) which the biomarker.R function calls. While there is code in the biomarker.R function to control for GE for LIN, we do not computer and provide this data.
+
+
+3. **Random Forest**: combines continuous and discrete genomic features in a multivariate analysis to predict the viabilities of cancer cell lines. The purpose of the random forest prediction is to assess feature importance in strong models. We consider valid models to have a model R2 > 0.2. There are two types of analyses run for random forest which vary based on input genomic features:
+ 1. CCLE model (x-ccle): LIN + MUT + GE + CNA - these are features present in almost all available cell lines
+ 2. All Model (x-all): x-ccle + PROT + XPR + miRNA + MET - all features present in a subset of cell lines
+
+The random forest module used is the R Ranger found in our cdsr_models [random_forest](https://github.com/broadinstitute/cdsr_models/blob/master/R/random_forest.R) which the biomarker.R function calls.
+
diff --git a/biomarker/aws_batch.sh b/biomarker/aws_batch.sh
new file mode 100755
index 0000000..09b5aac
--- /dev/null
+++ b/biomarker/aws_batch.sh
@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+
+# read in flagged arguments
+while test $# -gt 0; do
+ case "$1" in
+ -h| --help)
+ Rscript /biomarkers.R --help
+ exit 0
+ ;;
+ -c| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -b| --base_dir)
+ shift
+ base_dir=$1
+ ;;
+ -o| --out)
+ shift
+ out_dir=$1
+ ;;
+ -s | --project_key_start_index)
+ shift
+ PROJECT_KEY_START_INDEX=$1
+ ;;
+ -d| --biomarker_dir)
+ shift
+ biomarker_dir=$1
+ ;;
+ -f| --file)
+ shift
+ biomarker_file=$1
+ ;;
+ -q| --qc_file)
+ shift
+ qc_file=$1
+ ;;
+ -i| --failed_indicies)
+ shift
+ FAILED_INDICES=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+chmod +x /biomarkers.R
+chmod +x /src/biomarker_functions.R
+export HDF5_USE_FILE_LOCKING=FALSE
+
+echo FAILED_INDICES: "$FAILED_INDICES"
+#Case: Multiple Failed Jobs run from rerun-script as array job of size: n_failed_jobs
+if [[ -n "${AWS_BATCH_JOB_ARRAY_INDEX}" && -n $FAILED_INDICES ]]; then # -n is same as ! -z
+ #Case: Multiple Failed Jobs run from rerun-script as array job of size: n_failed_jobs
+ IFS=',' read -r -a failed_indices <<< "${FAILED_INDICES}"
+ RERUN_INDEX="${failed_indices[${AWS_BATCH_JOB_ARRAY_INDEX}]}"
+ echo "RERUN_INDEX IS: ${RERUN_INDEX}"
+elif [[ -n "$FAILED_INDICES" ]]; then
+ #case: Single failed job, cloned job and passed as parameter from command UI in AWS
+ IFS=',' read -r -a failed_indices <<< "${FAILED_INDICES}"
+ RERUN_INDEX="${failed_indices[0]}"
+ echo "RERUN_INDEX IS: ${RERUN_INDEX}"
+fi
+
+
+batch_index=0
+if [[ -n $projects ]]
+then
+ if [[ -n "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ echo "PROJECT_KEY_START_INDEX:" "$PROJECT_KEY_START_INDEX"
+ if [[ -n $RERUN_INDEX ]]; then
+ echo "RERUN_INDEX:" "${RERUN_INDEX}"
+ batch_index=$(($PROJECT_KEY_START_INDEX + ${RERUN_INDEX}))
+ else
+ echo "AWS_BATCH_JOB_ARRAY_INDEX:" "$AWS_BATCH_JOB_ARRAY_INDEX"
+ batch_index=$(($PROJECT_KEY_START_INDEX + ${AWS_BATCH_JOB_ARRAY_INDEX}))
+ fi
+ echo "BATCH INDEX:" $batch_index
+ fi
+ pert_id=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_id')
+ project=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ plate=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_plate')
+ biomarker_file=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].feature')
+ cleaned_pert_id=$(echo "${pert_id//|/$'_'}")
+ sanitized_pert_id="${cleaned_pert_id^^}"
+ base_dir="${base_dir}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"
+ out_dir="${out_dir}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"/biomarker
+fi
+
+echo "${base_dir}" "${out_dir}" "${biomarker_dir}" "${biomarker_file}" "${qc_file}"
+
+args=(
+ -b "${base_dir}"
+ -o "${out_dir}"
+)
+
+if [[ ! -z $biomarker_dir ]]
+then
+ args+=(-d "${biomarker_dir}")
+fi
+
+if [[ ! -z $biomarker_file ]]
+then
+ args+=(-f "${biomarker_file}")
+fi
+
+if [[ ! -z $qc_file ]]
+then
+ args+=(-q "${qc_file}")
+fi
+
+if [[ "$pert_id" == "DMSO" ]]
+then
+ echo "Skipping DMSO"
+else
+ echo Rscript /biomarkers.R "${args[@]}"
+ Rscript /biomarkers.R "${args[@]}"
+fi
+
+exit_code=$?
+
+echo "$exit_code"
+exit $exit_code
diff --git a/biomarker/biomarkers.R b/biomarker/biomarkers.R
new file mode 100644
index 0000000..ea5dace
--- /dev/null
+++ b/biomarker/biomarkers.R
@@ -0,0 +1,352 @@
+# Script to run biomarker analysis on MTS data
+# creates continuous associations, discrete associations, RF table, model table
+
+# import necessary libraries and functions
+suppressMessages(source("./src/biomarker_functions.R"))
+
+#---- Read arguments ----
+
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-b", "--base_dir", default="", help="Input directory.")
+parser$add_argument("-o", "--out", default=getwd(), help = "Output directory. Default is working directory.")
+parser$add_argument("-d", "--biomarker_dir", default="https://s3.amazonaws.com/biomarker.clue.io/features",
+ help="Directory containing biomarker files.")
+parser$add_argument("-f", "--biomarker_file", default=NULL, help="Name of biomarker file. Optional")
+parser$add_argument("-q", "--qc", default=NULL, help = "Path to QC file to be used as confounders")
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+
+base_dir <- args$base_dir
+out_dir <- args$out
+biomarker_dir <- args$biomarker_dir
+biomarker_file <- args$biomarker_file
+qc_path <- args$qc
+
+# data names/types for loading from taiga
+rf_data <- c("x-all", "x-ccle")
+discrete_data <- c("lin", "mut")
+linear_data <- c("ge", "xpr", "cna", "met", "mirna", "rep", "prot", "shrna")
+linear_names <- c("GE", "XPR", "CNA", "MET", "miRNA", "REP", "PROT", "shRNA")
+
+if (!is.null(biomarker_file) && !biomarker_file %in% c(rf_data, discrete_data, linear_data)) {
+ stop("Unknown biomarker file. Please try again.")
+}
+
+# make output directory
+if (!dir.exists(out_dir)) dir.create(out_dir, recursive = T)
+
+#---- LOAD THE DATA ----
+drc_path <- list.files(base_dir, pattern = "DRC_TABLE.csv", full.names = T)
+lfc_path <- list.files(base_dir, pattern = "LEVEL5_LFC_.*\\.csv$", full.names = T)
+
+# read dose-response if it's there
+if (length(drc_path == 1)) {
+ DRC <- data.table::fread(drc_path) %>%
+ dplyr::distinct(across(any_of(c("ccle_name", "culture", "pool_id", "varied_id", "varied_iname", "auc", "log2.ic50", "max_dose",
+ "added_compounds", "added_ids", "added_doses", "pert_time", "pert_plate")))) %>%
+ dplyr::group_by(across(any_of(c("varied_id", "varied_iname", "max_dose", "pert_time", "pert_plate",
+ "added_compounds", "added_ids", "added_doses")))) %>%
+ dplyr::mutate(n_ic50 = sum(is.finite(log2.ic50))) %>%
+ dplyr::ungroup() %>%
+ dplyr::mutate(log2.ic50 = ifelse((is.finite(auc) & !is.finite(log2.ic50)) & n_ic50 >= 10,
+ log2(3 * max_dose), log2.ic50),
+ log2.auc = log2(auc)) %>%
+ dplyr::select(-n_ic50, -max_dose) %>%
+ dplyr::select(-auc) %>%
+ tidyr::pivot_longer(cols = c("log2.auc", "log2.ic50"),
+ names_to = "pert_dose", values_to = "response") %>%
+ dplyr::mutate(pert_dose = as.character(pert_dose),pert_time = as.character(pert_time)) %>%
+ dplyr::filter(is.finite(response)) %>%
+ dplyr::rename(pert_iname = varied_iname, pert_id = varied_id)
+} else {
+ print("No DRC table found. Continuing with LFC only")
+ DRC <- tibble()
+}
+
+# always expect LFC table
+if (length(lfc_path) == 1) {
+ LFC <- data.table::fread(lfc_path) %>%
+ dplyr::distinct(across(any_of(c("ccle_name", "culture", "pool_id", "pert_id", "pert_time",
+ "pert_iname", "pert_dose", "pert_plate", "LFC", "LFC_cb")))) %>%
+ dplyr::mutate(pert_dose = as.character(pert_dose),pert_time = as.character(pert_time))
+
+ if ("LFC_cb" %in% colnames(LFC)) {
+ LFC %<>%
+ dplyr::rename(response = LFC_cb)
+ } else {
+ LFC %<>%
+ dplyr::rename(response = LFC)
+ }
+
+} else {
+ stop(paste("There are", length(lfc_path), "LFC files in the supplied directory. Please try again with 1."),
+ call. = FALSE)
+}
+
+# TO DO: update with new pipeline columns
+# read QC table if passed
+if (!is.null(qc_path) && file_test("-f", qc_path)) {
+ qc_table <- data.table::fread(qc_path) %>%
+ dplyr::filter(pass,
+ pert_plate %in% (LFC$pert_plate %>% unique())) %>%
+ dplyr::group_by(ccle_name) %>%
+ dplyr::summarise(dr = median(dr), ssmd = median(ssmd), nnmd = median(nnmd),
+ .groups = "drop") %>%
+ column_to_rownames("ccle_name") %>%
+ as.matrix()
+} else {
+ print("No SSMD table supplied or readable defaulting to no confounders")
+ qc_table <- NULL
+}
+
+# combine into large table
+all_Y <- dplyr::bind_rows(DRC, LFC)
+
+if (is.null(biomarker_file) || biomarker_file == "rep") {
+ rep_meta <- data.table::fread(paste0(biomarker_dir, "/rep_info.csv")) %>%
+ dplyr::select(column_name, name) %>%
+ dplyr::mutate(column_name = paste0("REP_", column_name))
+}
+
+# get lineage principal components to use as confounder
+if (is.null(biomarker_file) || biomarker_file == "ge") {
+ LIN_PCs <- data.table::fread(paste0(biomarker_dir, "/linPCA.csv")) %>%
+ column_to_rownames("V1") %>% as.matrix()
+ confounder_overlap <- intersect(rownames(LIN_PCs), rownames(qc_table))
+ if (!is.null(qc_table)) LIN_PCs <- cbind(LIN_PCs[confounder_overlap, ], qc_table[confounder_overlap, ])
+}
+
+runs <- all_Y %>%
+ dplyr::distinct(across(any_of(c("pert_iname", "pert_id", "pert_time", "pert_dose", "pert_plate",
+ "added_compounds", "added_ids", "added_doses"))))
+
+#---- LOOP THORUGH DATASETS AND DOSES ----
+
+# linear associations
+linear_table <- list(); ix <- 1
+for(feat in 1:length(linear_data)) {
+
+ # if specified only process for given file
+ if (!is.null(biomarker_file) && linear_data[feat] != biomarker_file) {
+ next
+ }
+
+ print(linear_data[feat])
+
+ # load feature set
+ X <- data.table::fread(paste0(biomarker_dir, "/", linear_data[feat], ".csv")) %>%
+ column_to_rownames("V1") %>% as.matrix()
+
+ # for each perturbation get results
+ for(i in 1:nrow(runs)) {
+ # filter down to current dose (run)
+ run <- runs[i,]
+ Y <- all_Y %>%
+ dplyr::inner_join(run)
+ y <- Y$response; names(y) <- Y$ccle_name
+ y <- y[is.finite(y)]
+
+ # get overlapping data
+ overlap <- dplyr::intersect(rownames(X), names(y))
+ y <- y[overlap]
+ if (!is.null(qc_table)) {
+ W <- qc_table[overlap, ]
+ } else {
+ W <- NULL
+ }
+
+ # if insufficient data, skip
+ if (length(y) < 10 | min(y) == max(y)) {
+ next
+ } else {
+ # calculate correlations
+ res.lin <- cdsrmodels::lin_associations(X[overlap,], y, W = W)
+ res.cor <- res.lin$res.table %>%
+ cbind(., rho=res.lin$rho[rownames(.),], q.val=res.lin$q.val[rownames(.),]) %>%
+ tibble::as_tibble() %>%
+ dplyr::rename(feature = ind.var, coef = rho) %>%
+ dplyr::arrange(q.val) %>%
+ dplyr::mutate(rank = 1:n(),
+ feature_type = linear_names[feat]) %>%
+ dplyr::filter(rank <= 1000 | q.val < 0.1) %>%
+ dplyr::bind_cols(run)
+
+ # for repurposing replace metadata
+ if (linear_data[feat] == "rep") {
+ res.cor %<>%
+ dplyr::left_join(rep_meta, by = c("feature" = "column_name")) %>%
+ dplyr::select(-feature) %>%
+ dplyr::rename(feature = name) %>%
+ dplyr::mutate(feature = paste("REP", feature, sep = "_"))
+ }
+
+ # append to output tables
+ linear_table[[ix]] <- res.cor; ix <- ix + 1
+ }
+ }
+
+ # gene expression with lineage as confounder - not currently computed in biomarker analysis
+ # if (linear_data[feat] == "ge") {
+ #
+ # # for each perturbation get results
+ # for(i in 1:nrow(runs)) {
+ # # filter down to current dose (run)
+ # run <- runs[i,]
+ # Y <- all_Y %>%
+ # dplyr::inner_join(run)
+ # y <- Y$response; names(y) <- Y$ccle_name
+ # y <- y[is.finite(y)]
+ #
+ # overlap <- dplyr::intersect(rownames(X), names(y)) %>%
+ # dplyr::intersect(., rownames(LIN_PCs))
+ # y <- y[overlap]
+ #
+ # if (length(y) < 10 | min(y) == max(y)) {
+ # next
+ # } else {
+ # # check that there are unique confounders
+ # if (all(apply(LIN_PCs[overlap,], 2, function(x) length(unique(x)) == 1))) {
+ # next
+ # } else {
+ # res.lin <- cdsrmodels::lin_associations(X[overlap,], y, W = LIN_PCs[overlap,])
+ # res.cor <- res.lin$res.table %>%
+ # cbind(., rho=res.lin$rho[rownames(.),], q.val=res.lin$q.val[rownames(.),]) %>%
+ # tibble::as_tibble() %>%
+ # dplyr::rename(feature = ind.var, coef = rho) %>%
+ # dplyr::arrange(q.val) %>%
+ # dplyr::mutate(rank = 1:n(),
+ # feature_type = "GE_noLIN") %>%
+ # dplyr::filter(rank <= 1000 | q.val < 0.1) %>%
+ # dplyr::bind_cols(run)
+ #
+ # linear_table[[ix]] <- res.cor; ix <- ix + 1
+ # }
+ # }
+ # }
+ # }
+}
+if (length(linear_table) > 0) {
+ linear_table %<>% dplyr::bind_rows()
+ file_suffix <- if (is.null(biomarker_file)) "" else paste0("_", biomarker_file)
+ file_name <- paste0(out_dir, "/continuous_associations", file_suffix, ".csv")
+ write.csv(linear_table, file_name, row.names=FALSE)
+}
+
+# repeat for discrete t-test
+discrete_table <- list(); ix <- 1
+for(feat in 1:length(discrete_data)) {
+ # if specified only process for given file
+ if (!is.null(biomarker_file) && discrete_data[feat] != biomarker_file) {
+ next
+ }
+
+ print(discrete_data[feat])
+
+ # load feature set
+ X <- data.table::fread(paste0(biomarker_dir, "/", discrete_data[feat], ".csv")) %>%
+ column_to_rownames("V1") %>% as.matrix()
+
+ for(i in 1:nrow(runs)) {
+ # filter down to current dose (run)
+ run <- runs[i,]
+ Y <- all_Y %>%
+ dplyr::inner_join(run)
+ y <- Y$response; names(y) <- Y$ccle_name
+ y <- y[is.finite(y)]
+
+ # get overlapping data
+ overlap <- dplyr::intersect(rownames(X), names(y))
+ y <- y[overlap]
+ if (!is.null(qc_table)) {
+ W <- qc_table[overlap, ]
+ } else {
+ W <- NULL
+ }
+
+ if (length(y) < 10 | min(y) == max(y)) {
+ next
+ } else {
+ res.disc <- cdsrmodels::discrete_test(X[overlap,], y, W = W) %>%
+ dplyr::mutate(feature_type = toupper(discrete_data[feat]))
+
+ res.disc %<>%
+ dplyr::bind_cols(run)
+
+ # only keep top 500 mutations
+ if (discrete_data[feat] == "mut" & nrow(res.disc) > 0) {
+ res.disc %<>%
+ dplyr::arrange(q.value) %>%
+ dplyr::mutate(rank = 1:n()) %>%
+ dplyr::filter(rank <= 500) %>%
+ dplyr::select(-rank)
+ }
+
+ discrete_table[[ix]] <- res.disc; ix <- ix + 1
+ }
+ }
+}
+if (length(discrete_table) > 0) {
+ discrete_table %<>% dplyr::bind_rows()
+ file_suffix <- if (is.null(biomarker_file)) "" else paste0("_", biomarker_file)
+ file_name <- paste0(out_dir, "/discrete_associations", file_suffix, ".csv")
+ write.csv(discrete_table, file_name, row.names=FALSE)
+}
+
+# repeat for random forest
+random_forest_table <- list(); model_table <- list(); ix <- 1
+for(feat in 1:length(rf_data)) {
+ # if specified only process for given file
+ if (!is.null(biomarker_file) && rf_data[feat] != biomarker_file) {
+ next
+ }
+ print(rf_data[feat])
+
+ # load feature set
+ X <- data.table::fread(paste0(biomarker_dir, "/", rf_data[feat], ".csv")) %>%
+ column_to_rownames("V1") %>% as.matrix()
+
+ model <- word(rf_data[feat], 2, sep = fixed("-"))
+
+ for (i in 1:nrow(runs)) {
+ run <- runs[i,]
+ Y <- all_Y %>%
+ dplyr::inner_join(run)
+ y <- Y$response; names(y) <- Y$ccle_name
+ y <- y[is.finite(y)]
+
+ # get overlapping data
+ overlap <- dplyr::intersect(rownames(X), names(y))
+ y <- y[overlap]
+ if (!is.null(qc_table)) {
+ W <- qc_table[overlap, ]
+ } else {
+ W <- NULL
+ }
+
+ if (length(y) < 10 | min(y) == max(y)) {
+ next
+ } else {
+ res.rf <- cdsrmodels::random_forest(X[overlap,], y, W = W)
+ res.model <- res.rf$model_table %>%
+ dplyr::distinct(MSE, MSE.se, R2, PearsonScore) %>%
+ dplyr::mutate(model = model) %>%
+ dplyr::bind_cols(run)
+ res.features <- res.rf$model_table %>%
+ dplyr::distinct(feature, RF.imp.mean, RF.imp.sd, RF.imp.stability, rank) %>%
+ dplyr::mutate(model = model) %>%
+ dplyr::bind_cols(run)
+ random_forest_table[[ix]] <- res.features; model_table[[ix]] <- res.model
+ ix <- ix + 1
+ }
+ }
+}
+if (length(random_forest_table) > 0) {
+ random_forest_table %<>% dplyr::bind_rows(); model_table %<>% dplyr::bind_rows()
+ file_suffix <- if (is.null(biomarker_file)) "" else paste0("_", biomarker_file)
+ model_name <- paste0(out_dir, "/model_table", file_suffix, ".csv")
+ rf_name <- paste0(out_dir, "/RF_table", file_suffix, ".csv")
+ write.csv(random_forest_table, rf_name, row.names=FALSE)
+ write.csv(model_table, model_name, row.names=FALSE)
+}
diff --git a/biomarker/build_docker.sh b/biomarker/build_docker.sh
new file mode 100755
index 0000000..9ad6f84
--- /dev/null
+++ b/biomarker/build_docker.sh
@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+docker build --platform linux/amd64 -t prismcmap/biomarker:production --rm=true .
diff --git a/biomarker/docker_push_tags.sh b/biomarker/docker_push_tags.sh
new file mode 100755
index 0000000..2928f0d
--- /dev/null
+++ b/biomarker/docker_push_tags.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+docker push prismcmap/biomarker:production
+
+# trigger
\ No newline at end of file
diff --git a/biomarker/docker_run.sh b/biomarker/docker_run.sh
new file mode 100755
index 0000000..ec4b3b8
--- /dev/null
+++ b/biomarker/docker_run.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+
+docker run --rm \
+--name bar2 \
+-v /Users/jasiedu/WebstormProjects/TEST_CPS017_P1000_NEWFMT3:/cmap/macchiato \
+-v /Users/jasiedu/WebstormProjects/biomarker/.cache/:/cmap/biomarker_cache \
+-e projects="$(cat /Users/jasiedu/WebstormProjects/macchiato/foo1.json)" \
+-e AWS_BATCH_JOB_ARRAY_INDEX=0 \
+-it cmap/biomarker-module:dev \
+-b /cmap/macchiato/projects \
+-o /cmap/macchiato/projects \
+-d "/cmap/biomarker_cache"
+
+
+
diff --git a/biomarker/src/biomarker_functions.R b/biomarker/src/biomarker_functions.R
new file mode 100644
index 0000000..2459b04
--- /dev/null
+++ b/biomarker/src/biomarker_functions.R
@@ -0,0 +1,9 @@
+# Helper functions for the initial processing step of the MTS pipeline
+
+#---- Load Packages ----
+# packages
+library(tidyverse)
+library(cdsrmodels)
+library(magrittr)
+library(argparse)
+library(readr)
diff --git a/biomarker_files/Dockerfile b/biomarker_files/Dockerfile
new file mode 100644
index 0000000..4aaeef5
--- /dev/null
+++ b/biomarker_files/Dockerfile
@@ -0,0 +1,3 @@
+FROM busybox
+
+CMD [ "ps", "faux" ]
\ No newline at end of file
diff --git a/biomarker_files/README.md b/biomarker_files/README.md
new file mode 100644
index 0000000..622b064
--- /dev/null
+++ b/biomarker_files/README.md
@@ -0,0 +1,23 @@
+# Biomarker Files
+
+This module downloads relevant biomarker feature files from [taiga](https://cds.team/taiga/) and which are also available from [DepMap](https://depmap.org/portal/download/all/) for non-Broad collaborators.
+
+## File info
+
+The files downloaded from taiga by `biomarker_tables.R` are equivalent to the following files:
+
+- `Achilles_gene_effect.csv` (CRISPR dependencies)
+- `CCLE_expression.csv` (gene expression)
+- `primary-screen-replicate-collapsed-logfold-change.csv` (repurposing)
+- `D2_Achilles_gene_dep_scores.csv` (shRNA)
+- `CCLE_metabolomics_20190502.csv` (metablomics)
+- `protein_quant_current_normalized.csv` (proteomics)
+- `CCLE_RPPA_20181003.csv` (RPPA)
+- `CCLE_miRNA_20181103.gct` (miRNA)
+- `CCLE_gene_cn.csv` (copy number)
+- `CCLE_mutations.csv` (mutations)
+- `sample_info.csv` (lineages)
+
+## Transformation
+
+Each file is then transformed into matrix form and the column headers are simplified with the data type appended to allow for combining datasets (e.g. gene expression columns become `GE_` and CRISPR columns become `XPR_`)
diff --git a/biomarker_files/biomarker_tables.R b/biomarker_files/biomarker_tables.R
new file mode 100644
index 0000000..ee149e9
--- /dev/null
+++ b/biomarker_files/biomarker_tables.R
@@ -0,0 +1,292 @@
+# load libraries
+library(tidyverse)
+library(taigr)
+library(readr)
+library(magrittr)
+library(celllinemapr) # probably should remove dependency
+
+# load the data (internal versions)
+# make sure that you use PUBLIC ones (for data sharing and publication reasons)
+# datasets that don't change run to run are labeled as stable in comment
+
+# shRNA (stable I think)
+shRNA <- load.from.taiga(data.name='demeter2-combined-dc9c',
+ data.version=19,
+ data.file='gene_means_proc')
+# micro RNA (stable)
+miRNA <- load.from.taiga(data.name='mirna-expression-2c5f',
+ data.version=3,
+ data.file='CCLE_miRNA_20180525')
+# metabolomics (stable)
+MET <- load.from.taiga(data.name='metabolomics-cd0c',
+ data.version=4,
+ data.file='CCLE_metabolomics_20190502')
+# RPPA (proteomics, stable)
+RPPA <- load.from.taiga(data.name='depmap-rppa-1b43',
+ data.version=3,
+ data.file='CCLE_RPPA_20181003')
+# damaging mutations
+damMUT <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='CCLE_mutations_bool_damaging')
+# hotspot mutations
+hsMUT <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='CCLE_mutations_bool_hotspot')
+# other mutations
+otherMUT <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='CCLE_mutations_bool_nonconserving')
+# lineages (extracted from sample_info file)
+LIN <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='sample_info')
+# CRISPR gene effect
+XPR <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='Achilles_gene_effect')
+# RNA expression
+GE <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='CCLE_expression')
+# copy number (not absolute)
+CNA <- load.from.taiga(data.name='public-21q4-a0d6',
+ data.version=8,
+ data.file='CCLE_gene_cn')
+# repurposing primary (stable)
+REP <- load.from.taiga(data.name='primary-screen-e5c7',
+ data.version=11,
+ data.file='primary-replicate-collapsed-logfold-change')
+# repurposing primary metadata (stable)
+rep_info <- load.from.taiga(data.name='primary-screen-e5c7',
+ data.version=11,
+ data.file='primary-replicate-collapsed-treatment-info')
+# proteomics (mass spec, stable I think)
+PROT <- load.from.taiga(data.name='total-proteome--5c50',
+ data.version=2,
+ data.file='normalized_protein_abundance')
+
+
+# PRISM cell lines (taken from a recent screen)
+cell_lines <- data.table::fread("~/Desktop/Old_MTS/MTS017_final/SSMD_TABLE.csv") %>%
+ dplyr::distinct(ccle_name) %>%
+ dplyr::mutate(broad_id = celllinemapr::ccle.to.arxspan(ccle_name, ignore.problems = T, check.unique.mapping = F)) %>%
+ dplyr::filter(!is.na(broad_id)) # need arxspan to join
+
+# for each table:
+# 1) rename columns (add feature abbreviation to front)
+# 2) rename rows by joining with PRISM cell table by arxspan (to get ccle_name)
+
+# shRNA
+shRNA <- t(shRNA) # transpose
+colnames(shRNA) <- paste("shRNA", word(colnames(shRNA), 1, sep = " "), sep = "_")
+shRNA <- as_tibble(shRNA, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# miRNA
+colnames(miRNA) <- paste("miRNA", colnames(miRNA), sep = "_")
+miRNA <- as_tibble(miRNA, rownames = "broad_id") %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# metabolomics
+colnames(MET) <- paste("MET", colnames(MET), sep = "_")
+MET <- as_tibble(MET, rownames = "broad_id") %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# protein expression
+colnames(RPPA) <- paste("RPPA", colnames(RPPA), sep = "_")
+RPPA <- as_tibble(RPPA, rownames = "broad_id") %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# lineage
+LIN %<>%
+ dplyr::rename(ccle_name = CCLE_Name, depmap_id = DepMap_ID) %>%
+ dplyr::filter(!grepl("MERGED", ccle_name)) # remove merged lines
+
+# lineage table (write to long form)
+lineages <- LIN %>%
+ dplyr::distinct(depmap_id, ccle_name, lineage, lineage_subtype) %>%
+ dplyr::filter(lineage != "unknown") # only keep known values
+all_annotations <- dplyr::union(lineages$lineage, lineages$lineage_subtype)
+
+# make abbreviations and add to long table
+abbreviations <- abbreviate(stringr::str_replace_all(all_annotations,
+ "[[:punct:]]", " "),
+ minlength = 4, strict = F,
+ method = "both.sides") %>%
+ toupper()
+names(abbreviations) <- all_annotations
+lineages$lin_abbreviation <- abbreviations[lineages$lineage]
+lineages$lin_sub_abbreviation <- abbreviations[lineages$lineage_subtype]
+readr::write_csv(lineages, "./lineages.csv") # write abbreviations file
+
+# create matrix with lineage, subtype, and sub-subtype (one hot encoded)
+lin <- LIN %>% dplyr::distinct(ccle_name, lineage) %>%
+ dplyr::filter(lineage != "")
+sub <- LIN %>% dplyr::distinct(ccle_name, lineage_subtype) %>%
+ dplyr::rename(lineage = lineage_subtype) %>%
+ dplyr::filter(lineage != "")
+sub2 <- LIN %>% dplyr::distinct(ccle_name, lineage_sub_subtype) %>%
+ dplyr::rename(lineage = lineage_sub_subtype) %>%
+ dplyr::filter(lineage != "")
+LIN_table <- dplyr::bind_rows(lin, sub, sub2) %>%
+ dplyr::distinct() %>%
+ dplyr::mutate(from = 1)
+LIN <- reshape2::acast(LIN_table, ccle_name ~ lineage,
+ value.var = "from", fill = 0)
+colnames(LIN) <- paste("LIN", colnames(LIN), sep = "_")
+
+# gene expression (RNA)
+colnames(GE) <- paste("GE", word(colnames(GE), 1, sep = " "), sep = "_")
+GE <- as_tibble(GE, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# gene effect (CRISPR)
+colnames(XPR) <- paste("XPR", word(colnames(XPR), 1, sep = " "), sep = "_")
+XPR <- as_tibble(XPR, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# copy number
+colnames(CNA) <- paste("CNA", word(colnames(CNA), 1, sep = " "), sep = "_")
+CNA <- as_tibble(CNA, rownames = "broad_id") %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# mutations (damaging, hotspot, other)
+colnames(damMUT) <- paste("MUT_dam", word(colnames(damMUT), 1, sep = " "), sep = "_")
+damMUT <- as_tibble(damMUT, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct()
+colnames(hsMUT) <- paste("MUT_hs", word(colnames(hsMUT), 1, sep = " "), sep = "_")
+hsMUT <- as_tibble(hsMUT, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct()
+colnames(otherMUT) <- paste("MUT_other", word(colnames(otherMUT), 1, sep = " "), sep = "_")
+otherMUT <- as_tibble(otherMUT, rownames = "broad_id", .name_repair = make.names) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct()
+MUT <- dplyr::inner_join(damMUT, hsMUT) %>%
+ dplyr::inner_join(otherMUT) %>%
+ column_to_rownames("ccle_name")
+
+# repurposing
+rep_info %<>%
+ dplyr::select(column_name, dose, name, screen_id) %>%
+ dplyr::filter(column_name %in% colnames(REP))
+colnames(REP) <- paste("REP", colnames(REP), sep = "_")
+REP <- as_tibble(REP, rownames = "broad_id") %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# total proteome
+colnames(PROT) <- paste("PROT", word(colnames(PROT), 1, sep = " "), sep = "_")
+PROT <- as_tibble(PROT, rownames = "broad_id",
+ .name_repair = function(x) make.names(x, unique = T)) %>%
+ dplyr::distinct() %>%
+ dplyr::inner_join(cell_lines) %>%
+ dplyr::select(-broad_id) %>%
+ dplyr::distinct() %>%
+ column_to_rownames("ccle_name")
+
+# filter out by variance > 0 (can alter threshold)
+MUT <- MUT[, apply(MUT, 2, function(x) var(x, na.rm = TRUE)) > 0]
+GE <- GE[, apply(GE, 2, function(x) var(x, na.rm = TRUE)) > 0]
+shRNA <- shRNA[, apply(shRNA, 2, function(x) var(x, na.rm = TRUE)) > 0]
+CNA <- CNA[, apply(CNA, 2, function(x) var(x, na.rm = TRUE)) > 0]
+XPR <- XPR[, apply(XPR, 2, function(x) var(x, na.rm = TRUE)) > 0]
+LIN <- LIN[, apply(LIN, 2, function(x) var(x, na.rm = TRUE)) > 0]
+miRNA <- miRNA[, apply(miRNA, 2, function(x) var(x, na.rm = TRUE)) > 0]
+RPPA <- RPPA[, apply(RPPA, 2, function(x) var(x, na.rm = TRUE)) > 0]
+MET <- MET[, apply(MET, 2, function(x) var(x, na.rm = TRUE)) > 0]
+PROT <- PROT[, apply(PROT, 2, function(x) var(x, na.rm = TRUE)) > 0]
+
+# write files
+write.csv(shRNA, "./data/shrna.csv")
+write.csv(CNA, "./data/cna.csv")
+write.csv(MUT, "./data/mut.csv")
+write.csv(GE, "./data/ge.csv")
+write.csv(LIN, "./data/lin.csv")
+write.csv(MET, "./data/met.csv")
+write.csv(miRNA, "./data/mirna.csv")
+write.csv(XPR, "./data/xpr.csv")
+write.csv(RPPA, "./data/rppa.csv")
+write.csv(REP, "./data/rep.csv")
+readr::write_csv(rep_info, "./data/rep_info.csv")
+write.csv(PROT, "./data/prot.csv")
+
+# make combined tables for multivariate analyses (only keeps complete rows)
+
+# ccle model = GE + LIN + MUT + CNA
+X.ccle <- GE %>% rownames_to_column(var = "ccle_name") %>%
+ dplyr::inner_join(as_tibble(LIN, rownames = "ccle_name")) %>%
+ dplyr::inner_join(MUT %>% rownames_to_column(var = "ccle_name")) %>%
+ dplyr::inner_join(CNA %>% rownames_to_column(var = "ccle_name")) %>%
+ column_to_rownames("ccle_name")
+
+# all model = ccle + XPR + RPPA + miRNA + MET
+X.all <- X.ccle %>% rownames_to_column(var = "ccle_name") %>%
+ dplyr::inner_join(XPR %>% rownames_to_column(var = "ccle_name")) %>%
+ dplyr::inner_join(RPPA %>% rownames_to_column(var = "ccle_name")) %>%
+ dplyr::inner_join(miRNA %>% rownames_to_column(var = "ccle_name")) %>%
+ dplyr::inner_join(MET %>% rownames_to_column(var = "ccle_name")) %>%
+ column_to_rownames("ccle_name")
+
+# remove any columns with NAs (ranger requirement)
+X.ccle <- X.ccle[, apply(X.ccle, 2, function(x) !any(is.na(x)))]
+X.all <- X.all[, apply(X.all, 2, function(x) !any(is.na(x)))]
+
+# write tables
+write.csv(X.all, "./data/x-all.csv")
+write.csv(X.ccle, "./data/x-ccle.csv")
+
+# run PCA on lineage table (for use as confounder)
+lin_PCs <- gmodels::fast.prcomp(LIN)
+lin_PCs <- LIN %*% lin_PCs$rotation[, lin_PCs$sdev > 0.2]
+write.csv(lin_PCs, "./data/linPCA.csv")
+
+# make long form mutation table (for reports)
+mutations <- MUT %>%
+ rownames_to_column("ccle_name") %>%
+ reshape2::melt(id.vars = "ccle_name",
+ variable.name = "mutation",
+ value.name = "is_mutant") %>%
+ dplyr::filter(is_mutant > 0) %>%
+ dplyr::select(-is_mutant) %>%
+ dplyr::mutate(mutation = word(mutation, 2, -1, sep = "_"))
+readr::write_csv(mutations, "./mutations.csv")
diff --git a/collate/.dockerignore b/collate/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/collate/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/collate/Dockerfile b/collate/Dockerfile
new file mode 100644
index 0000000..76b83c0
--- /dev/null
+++ b/collate/Dockerfile
@@ -0,0 +1,16 @@
+FROM cmap/base-merino:latest
+MAINTAINER Anup Jonchhe
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./collate_bash.sh /clue/bin/collate_bash
+COPY ./collate.py /clue/bin/collate.py
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+
+WORKDIR /
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["collate_bash"]
+
diff --git a/collate/README.md b/collate/README.md
new file mode 100644
index 0000000..6c72206
--- /dev/null
+++ b/collate/README.md
@@ -0,0 +1,82 @@
+# Collate Module
+
+## Collate Module Overview
+
+Collates assemble output, module takes in a project directory and searches for the following patterns:
+
+ `{PROJECT_DIR}/\*/assemble/\*/*_LEVEL2_MFI_*.gctx`
+
+ `{PROJECT_DIR}/\*/assemble/\*/*_LEVEL2_COUNT_*.gctx`
+
+
+The file structure :
+ ```
+ {PROJECT_DIR}/
+ {PLATE_DIR}/
+ assemble/
+ {PLATE_DIR}
+ *_LEVEL2_MFI_*.gctx
+ *_LEVEL2_COUNT_*.gctx
+ {PLATE_DIR}/
+ ```
+
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/collate) run:
+
+```
+docker pull prismcmap/collate:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `collate_bash.sh` which is a wrapper around `collate.py` for more information using python run
+
+```
+$ python collate.py --help
+usage: collate.py [-h] --proj_dir PROJ_DIR --cohort_name COHORT_NAME --build_dir BUILD_DIR [--search_pattern SEARCH_PATTERN]
+ [--verbose]
+
+optional arguments:
+ -h, --help show this help message and exit
+ --proj_dir PROJ_DIR, -pd PROJ_DIR
+ Required: Path to the pod directory you want to run
+ card on (default: None)
+ --cohort_name COHORT_NAME, -cn COHORT_NAME
+ Required: String designating the prefix to each build
+ file eg. PCAL075-126_T2B (default: None)
+ --build_dir BUILD_DIR, -bd BUILD_DIR
+ Required: outfolder for build files (default: None)
+ --search_pattern SEARCH_PATTERN, -sp SEARCH_PATTERN
+ Search for this string in the directory, only run
+ plates which contain it. Default is wildcard (default:
+ *)
+ --verbose, -v Whether to print a bunch of output (default: False)
+
+```
+
+### Example usage with Docker
+
+*Docker is the recommended method for running the collate module as it uses a deprecated version of python (Python 2.7).*
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+
+```
+docker run \
+ -it \
+ -v ~/{PROJECT_DIR}:/data/ \
+ prismcmap/collate:latest \
+ -pd /data/ -bd /data/build -cn {COHORT_NAME}
+```
+
+
+### Example usage with Python
+
+Running python requires an environment set up to use merino. See merino repo here: https://github.com/cmap/merino
+
+```
+python collate.py -pd {PROJECT_DIR} -bd {PROJECT_DIR}/build -cn {COHORT_NAME}
+```
diff --git a/collate/build_docker.sh b/collate/build_docker.sh
new file mode 100755
index 0000000..9db1a04
--- /dev/null
+++ b/collate/build_docker.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+REPO="prismcmap"
+NAME="collate"
+
+VERSION="v0.3.0" #change the version number for each new build
+TAG="latest"
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t $REPO/$NAME:$TAG -t $REPO/$NAME:$VERSION --rm=true .
+
+docker push $REPO/$NAME:$VERSION
+docker push $REPO/$NAME:$TAG
diff --git a/collate/collate.py b/collate/collate.py
new file mode 100644
index 0000000..e06ed76
--- /dev/null
+++ b/collate/collate.py
@@ -0,0 +1,214 @@
+import argparse
+import glob
+import logging
+import os
+import sys
+
+import cmapPy.pandasGEXpress.GCToo as GCToo
+import cmapPy.pandasGEXpress.concat as cg
+import cmapPy.pandasGEXpress.parse as pe
+import cmapPy.pandasGEXpress.write_gct as wg
+import cmapPy.pandasGEXpress.write_gctx as wgx
+import cmapPy.pandasGEXpress.subset_gctoo as sgt
+import merino.build_summary.ssmd_analysis as ssmd
+import merino.misc_tools.cut_to_l2 as cut_to_l2
+import merino.setup_logger as setup_logger
+import pandas as pd
+import numpy as np
+from math import floor, log10
+
+logger = logging.getLogger(setup_logger.LOGGER_NAME)
+
+
+def build_parser():
+
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ # The following arguments are required. These are files that are necessary for assembly and which change
+ # frequently between cohorts, replicates, etc.
+ parser.add_argument("--proj_dir", "-pd", help="Required: Path to the pod directory you want to run card on",
+ type=str, required=True)
+ parser.add_argument("--cohort_name", "-cn", help="Required: String designating the prefix to each build file eg. PCAL075-126_T2B",
+ type=str, required=True)
+ parser.add_argument("--build_dir", "-bd", help="Required: outfolder for build files",
+ type=str, required=True)
+ parser.add_argument("--search_pattern", "-sp",
+ help="Search for this string in the directory, only run plates which contain it. "
+ "Default is wildcard",
+ type=str, default='*', required=False)
+ parser.add_argument("--exclude_bcids", "-x",
+ help="Barcode Ids to include (LUAS or CTLBC) as comma-separated string"
+ "Default is none",
+ type=str, default=None, required=False)
+ parser.add_argument("--verbose", '-v', help="Whether to print a bunch of output", action="store_true", default=False)
+
+
+ return parser
+
+def build(search_pattern, cut=True, check_size=False):
+ gct_list = glob.glob(search_pattern)
+ old_len = len(gct_list)
+
+ if old_len == 0:
+ return None, None
+
+ if cut==True:
+ gct_list = cut_to_l2.cut_l1(gct_list)
+
+ new_len = len(gct_list)
+
+ logger.info('Number of old lysate plates removed = {}'.format(old_len - new_len))
+
+ if new_len == 0:
+ return None, None
+ gcts = []
+ failure_list = []
+ for gct in gct_list:
+ temp = pe.parse(gct)
+ gcts.append(temp)
+ if temp.data_df.shape[1] <= 349 and check_size == True:
+ failure_list.append(os.path.basename(gct).replace('_NORM.gct', ''))
+
+ for ct in gcts:
+ ct.row_metadata_df = gcts[0].row_metadata_df
+
+ fields_to_remove = [x for x in gcts[0].row_metadata_df.columns if
+ x in ['det_plate', 'det_plate_scan_time', 'assay_plate_barcode']]
+
+
+ concat_gct = cg.hstack(gcts, False, None, fields_to_remove=fields_to_remove)
+
+ concat_gct_wo_meta = GCToo.GCToo(data_df = concat_gct.data_df, row_metadata_df = pd.DataFrame(index=concat_gct.data_df.index),
+ col_metadata_df=pd.DataFrame(index=concat_gct.col_metadata_df.index))
+
+# logger.debug(outfile + 'n{}x{}'.format(concat_gct.data_df.shape[1], concat_gct.data_df.shape[0]) + file_suffix)
+
+ return concat_gct, failure_list
+
+
+def mk_gct_list(search_pattern):
+ #cut = False
+ gct_list = glob.glob(search_pattern)
+ old_len = len(gct_list)
+
+ if cut == True:
+ gct_list = cut_to_l2.cut_l1(gct_list)
+
+ new_len = len(gct_list)
+
+ print('Number of old lysate plates removed = {}'.format(old_len - new_len))
+
+ if new_len == 0:
+ return
+
+ return gct_list
+
+
+def mk_cell_metadata(args, failed_plates=None):
+ mfi_paths = glob.glob(os.path.join(args.proj_dir, args.search_pattern, 'assemble', args.search_pattern, '*MEDIAN.gct'))
+
+ cell_temp = pe.parse(mfi_paths[0])
+ cell_temp.row_metadata_df.to_csv(os.path.join(args.build_dir, args.cohort_name + '_cell_info.txt'), sep='\t')
+
+ if failed_plates:
+ # Calculate SSMD matrix using paths that were just grabbed and write out
+ ssmd_mat = ssmd.ssmd_matrix(cut_to_l2.cut_l1(paths))
+
+ ssmd_gct = GCToo.GCToo(data_df=ssmd_mat, col_metadata_df=pd.DataFrame(index=ssmd_mat.columns),
+ row_metadata_df=pd.DataFrame(index=ssmd_mat.index))
+ wg.write(ssmd_gct, os.path.join(args.build_dir, args.cohort_name + '_ssmd_matrix_n{}_{}.gct'.format(ssmd_gct.data_df.shape[1], ssmd_gct.data_df.shape[0])))
+
+ ssmd_failures = ssmd_gct.data_df.median()[ssmd_gct.data_df.median() < 2].index.tolist()
+ fails_dict = dict({'dropout_failures': failed_plates, 'ssmd_failures': ssmd_failures})
+ fails_df = pd.DataFrame(dict([(k, pd.Series(v)) for k, v in fails_dict.iteritems()]))
+ fails_df.to_csv(os.path.join(args.build_dir, 'failed_plates.txt'), sep='\t', index=False)
+
+
+def mk_inst_info(inst_data, args=None):
+
+ inst_info = inst_data.col_metadata_df
+ inst_info['profile_id'] = inst_info.index
+
+ for x in ['data_level', 'provenance']:
+ del inst_info[x]
+
+ inst_info.set_index('profile_id', inplace=True)
+
+ inst_info.to_csv(os.path.join(args.build_dir, args.cohort_name + '_inst_info.txt'), sep='\t')
+
+
+def write_gctx_with_dims(data, outfile):
+ logger.info("gct shape: {}".format(data.data_df.shape))
+ logger.info("gct shape: {}".format(data.data_df.shape))
+ path = outfile.rstrip('_')+'_n{}x{}'.format(data.data_df.shape[1], data.data_df.shape[0]) + '.gctx'
+ wgx.write(data, path)
+
+
+def remove_metadata_from_gctoo(gctoo):
+ return GCToo.GCToo(
+ data_df=gctoo.data_df,
+ row_metadata_df=pd.DataFrame(index=gctoo.data_df.index),
+ col_metadata_df=pd.DataFrame(index=gctoo.col_metadata_df.index)
+ )
+
+def remove_bcids(ds, bcids_to_remove):
+ rmeta = ds.row_metadata_df
+
+ rm_indexes = rmeta.loc[
+ rmeta.barcode_id.isin(bcids_to_remove)
+ ].index
+
+ print("removing BCIDS:" + ','.join(bcids_to_remove))
+ print("removing indexes:" + ','.join(rm_indexes))
+
+ return sgt.subset_gctoo(ds, exclude_rid=rm_indexes)
+
+
+
+def main(args):
+ search_pattern_dict = {
+ '*MEDIAN.gct': ['assemble', '_LEVEL2_MFI_'],
+ '*COUNT.gct': ['assemble', '_LEVEL2_COUNT_'],
+ }
+
+ data_dict = {}
+
+ for key in search_pattern_dict:
+ path = os.path.join(args.proj_dir, args.search_pattern, search_pattern_dict[key][0],
+ args.search_pattern,key)
+
+ out_path = os.path.join(args.build_dir, args.cohort_name + search_pattern_dict[key][1])
+
+ logger.info("working on {}".format(path))
+
+ data, _ = build(path, cut=True)
+ # exclude things here
+
+ if args.exclude_bcids:
+ bcids_to_remove = args.exclude_bcids.split(',')
+ logger.info("removing following barcode_ids: " + ",".join(bcids_to_remove))
+ data = remove_bcids(data, bcids_to_remove)
+
+ data_no_meta = remove_metadata_from_gctoo(data)
+ write_gctx_with_dims(data_no_meta, out_path)
+
+ data_dict[key] = data
+
+ mk_inst_info(data_dict['*MEDIAN.gct'], args=args)
+
+ try:
+ mk_cell_metadata(args, failure_list)
+ except Exception as e:
+ mk_cell_metadata(args)
+
+
+if __name__ == "__main__":
+ args = build_parser().parse_args(sys.argv[1:])
+ setup_logger.setup(verbose=args.verbose)
+
+ logger.debug("args: {}".format(args))
+
+ if not os.path.exists(args.build_dir):
+ logger.info("Making build directory: {}".format(args.build_dir))
+ os.mkdir(args.build_dir)
+
+ main(args)
diff --git a/collate/collate_bash.sh b/collate/collate_bash.sh
new file mode 100755
index 0000000..5057245
--- /dev/null
+++ b/collate/collate_bash.sh
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+
+print_help () {
+ printf "Usage ./collate_bash.sh [options]\nOptions include:\n"
+ printf -- "\t-pd, --proj_dir \t Path to the pod directory, assemble should be subfolder (required) \n"
+ printf -- "\t-bd, --build_dir \t Output folder for build files (required) \n"
+ printf -- "\t-cn, --cohort_name \t String designating the prefix to each build file (required)\n"
+ printf -- "\t-sp, --search_pattern \t Search string in proj_dir, only run matching plates, default is wildcard '*' \n"
+ printf -- "\t-x, --exclude_bcids \t Barcode Ids to include (LUAS or CTLBC) as comma-separated string. Default is none \n"
+ printf -- "\t-v, --verbose \t\t Verbose flag, print additional output \n"
+ printf -- "\t-h, --help \t\t Print this help text\n"
+}
+
+#optional
+if test $# -lt 1; then
+ print_help
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit 0
+ ;;
+ -pd| --proj_dir)
+ shift
+ PROJ_DIR=$1
+ ;;
+ -bd|--build_dir)
+ shift
+ BUILD_DIR=$1
+ ;;
+ -cn| --cohort_name)
+ shift
+ COHORT_NAME=$1
+ ;;
+ -sp| --search_pattern)
+ shift
+ SEARCH_PATTERN=$1
+ ;;
+ -x| --exclude_bcids)
+ shift
+ EXCLUDE_BCIDS=$1
+ ;;
+ -v| --verbose)
+ shift
+ VERBOSE=true
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+if [ ! -d $BUILD_DIR ]
+then
+ mkdir -p $BUILD_DIR
+fi
+
+##Run Collate
+
+if [[ -z $PROJ_DIR || -z $BUILD_DIR || -z $COHORT_NAME ]]
+then
+ printf "Required arguments missing\n"
+ exit -1
+fi
+
+args=(
+ -pd "$PROJ_DIR"
+ -bd "$BUILD_DIR"
+ -cn "$COHORT_NAME"
+)
+
+
+if [[ ! -z $SEARCH_PATTERN ]]
+then
+ args+=(-sp "$SEARCH_PATTERN")
+fi
+
+if [[ ! -z $EXCLUDE_BCIDS ]]
+then
+ args+=(-x "$EXCLUDE_BCIDS")
+fi
+
+if [[ ! -z $VERBOSE ]]
+then
+ args+=(-v)
+fi
+
+#setup environment
+source activate merino
+cd /cmap
+git clone https://github.com/cmap/merino.git
+cd /cmap/merino/
+python setup.py develop
+
+#return to /
+cd /
+
+#echo python collate.py "${args[@]}"
+python /clue/bin/collate.py "${args[@]}"
+exit_code=$?
+
+source deactivate
+exit $exit_code
diff --git a/collate/docker_push_tags.sh b/collate/docker_push_tags.sh
new file mode 100755
index 0000000..0a48959
--- /dev/null
+++ b/collate/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/collate:v0.1.0
+docker push prismcmap/collate:latest
+
+# trigger
\ No newline at end of file
diff --git a/compound_summary/Dockerfile b/compound_summary/Dockerfile
new file mode 100644
index 0000000..04b5dd2
--- /dev/null
+++ b/compound_summary/Dockerfile
@@ -0,0 +1,16 @@
+FROM cmap/base-clue-mts
+MAINTAINER Anup Jonchhe
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./src/metric_functions.R /src/metric_functions.R
+COPY ./compound_summary.R /compound_summary.R
+COPY ./aws_batch.sh /clue/bin/aws_batch
+RUN chmod -R +x /*.R
+
+WORKDIR /
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/compound_summary/README.md b/compound_summary/README.md
new file mode 100644
index 0000000..81538cd
--- /dev/null
+++ b/compound_summary/README.md
@@ -0,0 +1,59 @@
+# Compound Summary Module
+
+## Compound Summary Module Overview
+
+Calculate summary metrics for compounds in a project. The module takes in a project directory and searches for the following patterns:
+
+ - `LEVEL4_LFC_{COMBAT}`
+ - `discrete_associations.csv`
+ - `continuous_associations.csv`
+ - `model_table.csv`
+ - `RF_table.csv`
+
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/compound_summary) run:
+
+```
+docker pull prismcmap/compound_summary:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `aws_batch.sh` which is a wrapper around `compound_summary.R` for more information using R run
+
+```
+Rscript compound_summary.R --help
+usage: compound_summary.R [-h] [-i INPUT_DIR] [-o OUT]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -i INPUT_DIR, --input_dir INPUT_DIR
+ Input directory
+ -o OUT, --out OUT Output directory
+```
+
+### Example usage with R
+
+R execution requires installing the correct R packages which are outlined in the `docker_base` module.
+
+```
+Rscript compound_summary.R -i ~/Desktop/clue_data/project/data -o ~/Desktop/mts_results/project/data
+```
+
+### Example usage with Docker
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+```
+docker run \
+ -it \
+ -v ~/Desktop/clue_data/project/data:/in_data \
+ -v ~/Desktop/mts_results/project/data:/out_data \
+ prismcmap/compound_summary:latest \
+ -i /in_data \
+ -o /out_data
+```
diff --git a/compound_summary/aws_batch.sh b/compound_summary/aws_batch.sh
new file mode 100755
index 0000000..9d12b72
--- /dev/null
+++ b/compound_summary/aws_batch.sh
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+
+#return to /
+cd /
+
+print_help () {
+ Rscript /compound_summary.R --help
+}
+
+#optional
+if test $# -lt 1; then
+ print_help
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit 0
+ ;;
+ -f| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -i| --input_dir)
+ shift
+ DATA_PATH=$1
+ ;;
+ -o|--out)
+ shift
+ OUT_DIR=$1
+ ;;
+ -v| --verbose)
+ shift
+ VERBOSE=true
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+
+
+##Run Collate
+if [[ -z $DATA_PATH || -z $OUT_DIR ]]
+then
+ printf "Required arguments missing\n"
+ exit -1
+fi
+
+
+
+if [[ -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ batch=0
+else
+ batch=${AWS_BATCH_JOB_ARRAY_INDEX}
+fi
+
+PROJECT_DIR="${DATA_PATH}"
+
+if [[ ! -z $projects ]]
+then
+ PROJECT=$(cat "${projects}" | jq -r --argjson index ${batch} '.[$index].x_project_id')
+ PROJECT_DIR="${DATA_PATH}"/"${PROJECT,,}"/"${PROJECT^^}"/data/
+ OUT_DIR="${OUT_DIR}"/"${PROJECT,,}"/"${PROJECT^^}"/data/
+fi
+
+args=(
+ -i "$PROJECT_DIR"
+ -o "$OUT_DIR"
+)
+
+if [ ! -d $OUT_DIR ]
+then
+ mkdir -p $OUT_DIR
+fi
+
+
+if [[ ! -z $VERBOSE ]]
+then
+ args+=(-v)
+fi
+
+echo Rscript /compound_summary.R "${args[@]}"
+Rscript /compound_summary.R "${args[@]}"
+
+exit_code=$?
+exit $exit_code
diff --git a/compound_summary/build_docker.sh b/compound_summary/build_docker.sh
new file mode 100755
index 0000000..394c7c5
--- /dev/null
+++ b/compound_summary/build_docker.sh
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+docker build --platform linux/amd64 -t prismcmap/compound_summary:latest --rm=true .
+
diff --git a/compound_summary/compound_summary.R b/compound_summary/compound_summary.R
new file mode 100644
index 0000000..e7a517f
--- /dev/null
+++ b/compound_summary/compound_summary.R
@@ -0,0 +1,163 @@
+suppressMessages(library(tidyverse))
+suppressMessages(library(magrittr))
+suppressMessages(library(useful))
+suppressMessages(library(argparse))
+suppressMessages(source("./src/metric_functions.R"))
+
+#---- Read arguments ----
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-i", "--input_dir", default="", help="Input directory with one level 4 LFC file")
+parser$add_argument("-o", "--out", default="", help="Output directory")
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+input_dir <- args$input_dir
+out_dir <- args$out
+dir.create(out_dir, showWarnings = FALSE)
+
+
+lfc_files <- list.files(input_dir,pattern=("LEVEL4_LFC.*\\.csv$"), full.names=T)
+combat_file = lfc_files[str_detect(lfc_files, "COMBAT")]
+
+if (length(combat_file) == 1) {
+ lfc_file = combat_file
+} else if (length(lfc_files) == 1) {
+ lfc_file <- lfc_files[[1]]
+} else{
+ stop(paste("Issue with LFC files found in directory: \n", toString(lfc_files)),
+ call. = FALSE)
+}
+
+continous_files <- list.files(input_dir,pattern=("continuous_associations.csv$"), full.names=T)
+discrete_files <- list.files(input_dir,pattern=("discrete_associations.csv$"), full.names=T)
+random_forest.model_files <- list.files(input_dir,pattern=("model_table.csv$"), full.names=T)
+random_forest.feats_files <- list.files(input_dir,pattern=("RF_table.csv$"), full.names=T)
+
+# load the viability data ----
+
+lfc <- data.table::fread(lfc_file) %>%
+ dplyr::mutate(pert_dose = as.character(as.numeric(pert_dose)))
+
+# load the biomarker data ----
+
+continous <- data.table::fread(continous_files) %>%
+ dplyr::mutate(pert_dose = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ pert_dose, as.character(as.numeric(pert_dose))))
+discrete <- data.table::fread(discrete_files) %>%
+ dplyr::mutate(pert_dose = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ pert_dose, as.character(as.numeric(pert_dose))))
+
+random_forest.model <- random_forest.feats <- data.table::fread(random_forest.model_files) %>%
+ dplyr::mutate(pert_dose = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ pert_dose, as.character(as.numeric(pert_dose))))
+
+random_forest.feats <- random_forest.feats <- data.table::fread(random_forest.feats_files) %>%
+ dplyr::mutate(pert_dose = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ pert_dose, as.character(as.numeric(pert_dose))))
+
+# create the biomarker table first ----
+
+RF.TABLE <- random_forest.feats %>%
+ dplyr::filter(rank < 4) %>%
+ dplyr::group_by(pert_iname, pert_id, pert_plate, pert_dose, model) %>%
+ dplyr::arrange(rank) %>%
+ dplyr::mutate(feature = paste0(feature, " (", round(RF.imp.mean, 3), ")")) %>%
+ dplyr::summarise(feats = paste0(feature, collapse = ", ")) %>%
+ dplyr::ungroup() %>%
+ dplyr::left_join(random_forest.model %>%
+ dplyr::distinct(model, pert_iname, pert_id, pert_plate, pert_dose, PearsonScore)) %>%
+ dplyr::mutate(model1 = paste0("RF.feats.", model),
+ model = paste0("RF.score.", model)) %>%
+ tidyr::pivot_wider(names_from = "model", values_from = "PearsonScore") %>%
+ tidyr::pivot_wider(names_from = "model1", values_from = "feats") %>%
+ dplyr::group_by(pert_iname, pert_id, pert_plate, pert_dose) %>%
+ dplyr::summarise_all(function(x) x[!is.na(x)]) %>%
+ dplyr::ungroup()
+
+
+CONT.TABLE <- continous %>%
+ dplyr::filter(q.val < 0.05, abs(coef) > .1) %>%
+ dplyr::group_by(pert_iname, pert_id, pert_plate, pert_dose, feature_type) %>%
+ dplyr::arrange(coef) %>% dplyr::mutate(ix = ifelse(n() > 0, 1:n(), NA)) %>%
+ dplyr::arrange(-coef) %>% dplyr::mutate(ix = pmin(ix,1:n())) %>%
+ dplyr::filter(ix < 4) %>%
+ dplyr::arrange(coef) %>%
+ dplyr::mutate(feature = paste0(word(feature,2,-1,sep = fixed("_"))," (",round(coef,2), ")")) %>%
+ dplyr::summarise(feats = paste0(feature, collapse = ", ")) %>%
+ dplyr::ungroup() %>%
+ tidyr::pivot_wider(names_from = "feature_type",
+ values_from = "feats") %>%
+ dplyr::ungroup()
+
+DISC.TABLE <- discrete %>%
+ dplyr::filter(q.value < 0.05, abs(effect_size) > .1) %>%
+ dplyr::group_by(pert_iname, pert_id, pert_plate, pert_dose, feature_type) %>%
+ dplyr::arrange(effect_size) %>% dplyr::mutate(ix = ifelse(n() > 0, 1:n(), NA)) %>%
+ dplyr::arrange(-effect_size) %>% dplyr::mutate(ix = pmin(ix,1:n())) %>%
+ dplyr::filter(ix < 4) %>%
+ dplyr::arrange(effect_size) %>%
+ dplyr::mutate(feature = paste0(word(feature,2,-1,sep = fixed("_"))," (",round(effect_size,2), ")")) %>%
+ dplyr::summarise(feats = paste0(feature, collapse = ", ")) %>%
+ dplyr::ungroup() %>%
+ tidyr::pivot_wider(names_from = "feature_type",
+ values_from = "feats") %>%
+ dplyr::ungroup()
+
+
+BIOMARKER.TABLE <- RF.TABLE %>%
+ dplyr::full_join(CONT.TABLE) %>%
+ dplyr::full_join(DISC.TABLE)
+
+
+# Next create summary statistics -----
+
+
+summarize.lfc = function(ccle_name, pert_well, replicate, LFC_cb){
+ AverageRepCor = tibble(ccle_name, replicate, pert_well, LFC_cb) %>%
+ reshape2::acast(ccle_name ~ replicate + pert_well, value.var = "LFC_cb") %>%
+ cor(use = "p") %>%
+ reshape2::melt() %>%
+ dplyr::filter(Var1 != Var2) %>%
+ dplyr::summarise(m = mean(value, na.rm = T), n = n()/2)
+
+ res = tibble(ccle_name, LFC_cb) %>%
+ dplyr::group_by(ccle_name) %>%
+ dplyr::summarise(X = median(LFC_cb, na.rm = T)) %>%
+ dplyr::summarise(BimodalityCoefficient = bimodality_coefficient(X[is.finite(X)]),
+ CellLinesKilled = sum(X < log2(.3), na.rm = T)) %>%
+ dplyr::mutate(n.rep = AverageRepCor$n,
+ AverageRepCor = AverageRepCor$m)
+
+ return(res)
+}
+
+LFC.TABLE <- lfc %>%
+ dplyr::group_by(x_project_id, screen, pert_iname, pert_id, pert_plate, pert_dose,
+ ccle_name, pert_well, replicate) %>%
+ dplyr::summarise(LFC_cb = median(LFC_cb, na.rm = T)) %>%
+ dplyr::group_by(x_project_id, screen, pert_iname, pert_id, pert_plate, pert_dose) %>%
+ dplyr::summarise(summarize.lfc(ccle_name, pert_well, replicate, LFC_cb)) %>%
+ dplyr::ungroup()
+
+
+# Join and write to csv ----
+LFC.TABLE %>%
+ dplyr::full_join(BIOMARKER.TABLE) %>%
+ dplyr::group_by(pert_id, pert_iname, pert_plate) %>%
+ dplyr::mutate(BimodalityCoefficient = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ mean(BimodalityCoefficient, na.rm = T), BimodalityCoefficient),
+ n.rep = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ mean(n.rep, na.rm = T), n.rep),
+ AverageRepCor = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ mean(AverageRepCor, na.rm = T), AverageRepCor),
+ CellLinesKilled = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ mean(CellLinesKilled, na.rm = T), CellLinesKilled),
+ x_project_id = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ x_project_id[!is.na(x_project_id)][1], x_project_id),
+ screen = ifelse(pert_dose %in% c("log2.auc", "log2.ic50"),
+ screen[!is.na(screen)][1], screen)) %>%
+ dplyr::ungroup() %>%
+ write_csv(paste0(out_dir, "/compound_summary_metrics.csv"))
+
+
diff --git a/compound_summary/src/metric_functions.R b/compound_summary/src/metric_functions.R
new file mode 100644
index 0000000..a397ba5
--- /dev/null
+++ b/compound_summary/src/metric_functions.R
@@ -0,0 +1,42 @@
+library(tidyverse)
+library(magrittr)
+library(useful)
+
+# auxilary functions ----
+skewness<-function(x, finite=TRUE){
+ n=length(x)
+ S=(1/n)*sum((x-mean(x))^3)/(((1/n)*sum((x-mean(x))^2))^1.5)
+ if(finite==FALSE){
+ S=S
+ }else{
+ S=S*(sqrt(n*(n-1)))/(n-2)
+ }
+ return(S)
+}
+kurtosis<-function(x, finite){
+ n=length(x)
+ K=(1/n)*sum((x-mean(x))^4)/(((1/n)*sum((x-mean(x))^2))^2) - 3
+ if(finite==FALSE){
+ K=K
+ }
+ else{
+ K=((n-1)*((n+1)*K - 3*(n-1))/((n-2)*(n-3))) +3
+ }
+ return(K)
+}
+bimodality_coefficient<-function(x, finite=TRUE,...){
+ if(finite==TRUE){
+ G=skewness(x,finite)
+ sample.excess.kurtosis=kurtosis(x,finite)
+ K=sample.excess.kurtosis
+ n=length(x)
+ B=((G^2)+1)/(K+ ((3*((n-1)^2))/((n-2)*(n-3))))
+ }
+ else{
+ G=skewness(x,FALSE)
+ K=kurtosis(x,FALSE)
+ B=((G^2)+1)/(K)
+ }
+ return(B)
+}
+
diff --git a/convert-projectkeys-csv-to-json/.dockerignore b/convert-projectkeys-csv-to-json/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/convert-projectkeys-csv-to-json/Dockerfile b/convert-projectkeys-csv-to-json/Dockerfile
new file mode 100644
index 0000000..fa4e65f
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/Dockerfile
@@ -0,0 +1,14 @@
+FROM amazon/aws-lambda-nodejs:14
+COPY csv2json.sh /clue/bin/csv2json
+
+WORKDIR /
+COPY index.js ./
+COPY package*.json ./
+
+
+RUN npm install
+
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["csv2json"]
diff --git a/convert-projectkeys-csv-to-json/README b/convert-projectkeys-csv-to-json/README
new file mode 100644
index 0000000..c4ffa35
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/README
@@ -0,0 +1,5 @@
+This module converts compound keys from csv format to json so that
+downstream modules can read the JSON file from the file system.
+This will help get around the 8192 character limit for the container overrides
+portion of AWS Batch Jobs, since we do not have to pass the full
+compound keys as environment variables to each module
diff --git a/convert-projectkeys-csv-to-json/build_docker.sh b/convert-projectkeys-csv-to-json/build_docker.sh
new file mode 100755
index 0000000..3989165
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/build_docker.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/csv2json:develop \
+-t prismcmap/csv2json:v0.3.0 \
+--rm=true .
diff --git a/convert-projectkeys-csv-to-json/csv2json.sh b/convert-projectkeys-csv-to-json/csv2json.sh
new file mode 100755
index 0000000..2ab2807
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/csv2json.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+
+while test $# -gt 0; do
+ case "$1" in
+ -f| --compound_key_file)
+ shift
+ COMPOUND_KEY=$1
+ ;;
+ -o|--out)
+ shift
+ COMPOUND_KEY_JSON=$1
+ ;;
+ -l|--levels)
+ shift
+ LEVELS=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+args=(-f $COMPOUND_KEY_JSON)
+if [[ ! -z $LEVELS ]]
+then
+ args+=(-l $LEVELS)
+fi
+
+if [[ ! -z $COMPOUND_KEY && ! -z $COMPOUND_KEY_JSON ]]
+then
+ npx csvtojson $COMPOUND_KEY > $COMPOUND_KEY_JSON
+ node ./index.js "${args[@]}"
+
+else
+ echo "The full path to both compound key file and output json file must be specified"
+ exit 1
+fi
+
+exit_code=$?
+exit $exit_code
diff --git a/convert-projectkeys-csv-to-json/docker_push_tags.sh b/convert-projectkeys-csv-to-json/docker_push_tags.sh
new file mode 100755
index 0000000..eefd621
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/csv2json:v0.2.1
+docker push prismcmap/csv2json:latest
+
+# triggers
\ No newline at end of file
diff --git a/convert-projectkeys-csv-to-json/docker_run.sh b/convert-projectkeys-csv-to-json/docker_run.sh
new file mode 100755
index 0000000..682d944
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/docker_run.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+docker run --rm \
+--name bar2 \
+-v \
+/Users/naim/Documents/Work/Troubleshooting/SUSHI-MTS/:/data/ \
+-it prismcmap/csv2json:develop \
+-f /data/PMTS022_PR500_compound_key.csv \
+-o /data/PMTS022_PR500_compound_key.json \
+-l LEVEL4_LFC,LEVEL4_LFC_COMBAT,LEVEL5_LFC,LEVEL5_LFC_COMBAT
\ No newline at end of file
diff --git a/convert-projectkeys-csv-to-json/index.js b/convert-projectkeys-csv-to-json/index.js
new file mode 100644
index 0000000..ce28c55
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/index.js
@@ -0,0 +1,307 @@
+const _ = require("underscore");
+const { ArgumentParser } = require('argparse');
+const fsPromises = require("fs/promises");
+const Buffer = require("buffer");
+
+const LEVELS = [
+ 'inst_info',
+ 'cell_info',
+ 'QC_TABLE',
+ 'LEVEL2_COUNT',
+ 'LEVEL2_MFI',
+ 'LEVEL3_LMFI',
+ 'LEVEL4_LFC',
+ 'LEVEL4_LFC_COMBAT',
+ 'LEVEL5_LFC',
+ 'LEVEL5_LFC_COMBAT'
+];
+
+/**
+ *
+ * @param fileName
+ * @param projects
+ * @returns {Promise<*>}
+ */
+const writeOutput = async function(fileName,projects){
+ return await fsPromises.writeFile(fileName,JSON.stringify(projects));
+}
+
+const getProjectsWithCombinations = function(projectKeys){
+ return _.pluck(_.uniq(projectKeys.filter(function(projectKey){
+ return projectKey.is_combination === '1';
+ }), function (projectKey) {
+ return projectKey.x_project_id;
+ }), "x_project_id");
+}
+
+/**
+ *
+ * @param projectKeys
+ * @returns {*}
+ */
+const uniqueProjects= function(projectKeys){
+ const projectsWithCombinations = getProjectsWithCombinations(projectKeys);
+
+ const out = _.uniq(projectKeys, function (projectKey) {
+ return projectKey.x_project_id;
+ })
+
+ out.forEach((project) => {
+ if (projectsWithCombinations.includes(project.x_project_id)) {
+ project.combination_project = '1'
+ } else {
+ project.combination_project = '0'
+ }
+ })
+
+ return out;
+}
+
+const uniquePertPlates = function (projectKeys) {
+ return _.uniq(projectKeys, function (projectKey) {
+ return projectKey.pert_plate;
+ });
+}
+
+const detectCombinations = function (projectKeys) {
+ return projectKeys.forEach((compound) => {
+ //check if compound.pert_iname has | character
+ if (compound.pert_iname.includes("|")) {
+ compound.is_combination = '1'
+ } else{
+ compound.is_combination = '0'
+ }
+ })
+}
+
+const uniqueProjectKeysWithPertPlates = async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_uniq_pert_plates.json");
+ const projectsPertPlates = _.map(
+ _.uniq(projectKeys, function (projectKey) {
+ return [projectKey.x_project_id,projectKey.pert_plate].join('_');
+ }),
+ function(key) {
+ return { x_project_id: key.x_project_id, pert_plate: key.pert_plate };
+ }
+ );
+ return await writeOutput(outPath,projectsPertPlates);
+}
+/**
+ *
+ * @param projectKeys
+ * @param fileName
+ * @returns {Promise<*>}
+ */
+const uniques= async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_uniques.json");
+ const uniqueProjectKeys = uniqueProjects(projectKeys);
+ return await writeOutput(outPath,uniqueProjectKeys);
+}
+/**
+ *
+ * @param projectKeys
+ * @param fileName
+ * @returns {Promise<*>}
+ */
+const levels = async function(projectKeys,fileName, LEVELS){
+ const outPath = fileName.replace(".json","_levels.json");
+ const uniqueProjectKeys = uniqueProjects(projectKeys)
+ const projects = [];
+ for (let index = 0; index < uniqueProjectKeys.length; index++) {
+ const currentProject = uniqueProjectKeys[index];
+ for (let index1 = 0; index1 < LEVELS.length; index1++) {
+ const level = LEVELS[index1];
+ const project = {x_project_id: currentProject.x_project_id, level: level};
+ projects.push(project);
+ }
+ }
+ return await writeOutput(outPath,projects);
+}
+/**
+ *
+ * @param projectKeys
+ * @param fileName
+ * @returns {Promise<*>}
+ */
+const features = async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_features.json");
+ const features = [
+ "x-all",
+ "x-ccle",
+ "lin",
+ "mut",
+ "ge",
+ "xpr",
+ "cna",
+ "met",
+ "mirna",
+ "rep",
+ "prot",
+ "shrna"
+ ];
+
+ const projects = [];
+ for (let index = 0; index < projectKeys.length; index++) {
+ const project = projectKeys[index];
+ for (let index1 = 0; index1 < features.length; index1++) {
+ const feature = features[index1];
+ const clonedProject = Object.assign({}, project);
+ clonedProject.feature = feature;
+ projects.push(clonedProject);
+ }
+ }
+
+ const sortedProjectKeys = _.sortBy(projects, function(proj){
+ switch (proj.feature){
+ case 'lin':
+ return 1
+ case 'mut':
+ return 1
+ case 'ge':
+ return 1
+ case 'xpr':
+ return 1
+ case 'cna':
+ return 1
+ case 'met':
+ return 1
+ case 'mirna':
+ return 1
+ case 'rep':
+ return 1
+ case 'prot':
+ return 1
+ case 'shrna':
+ return 1
+ case 'x-all':
+ return 2
+ case 'x-ccle':
+ return 2
+ }
+ });
+
+
+ return await writeOutput(outPath,sortedProjectKeys);
+}
+/**
+ *
+ * @param patterns
+ * @param projectKeys
+ * @returns {[]}
+ */
+const searchPatterns = function(patterns,projectKeys){
+ const projects = [];
+ for (let index = 0; index < projectKeys.length; index++) {
+ const project = projectKeys[index];
+ for (let index1 = 0; index1 < patterns.length; index1++) {
+ const pattern = patterns[index1];
+ const clonedProject = Object.assign({}, project);
+ clonedProject.pattern = pattern;
+ projects.push(clonedProject);
+ }
+ }
+ return projects;
+}
+/**
+ *
+ * @param projectKeys
+ * @param fileName
+ * @returns {Promise<*>}
+ */
+const uniqueProjectsWithSearch = async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_proj_search_pattern.json");
+ const uniqueProjectKeys = uniqueProjects(projectKeys);
+ const patterns = [
+ "continuous_associations.csv",
+ "discrete_associations.csv",
+ "DRC_TABLE.csv",
+ "model_table.csv",
+ "RF_table.csv"
+ ];
+ const projects = searchPatterns(patterns,uniqueProjectKeys);
+ return await writeOutput(outPath,projects);
+}
+const uniqueProjectsWithCombinationSearch = async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_comb_search_pattern.json");
+ const combinationsOnly = projectKeys.filter(function(projectKey){
+ return projectKey.is_combination === '1';
+ })
+ const uniqueProjectKeys = uniqueProjects(combinationsOnly);
+ const patterns = [
+ "synergy_table.csv",
+ "bliss_mss_table.csv"
+ ];
+ const projects = searchPatterns(patterns,uniqueProjectKeys);
+ return await writeOutput(outPath,projects);
+}
+/**
+ *
+ * @param projectKeys
+ * @param fileName
+ * @returns {Promise<*>}
+ */
+const searchProjectPatterns = async function(projectKeys,fileName){
+ const outPath = fileName.replace(".json","_search_pattern.json");
+ const patterns = [
+ "discrete_associations*",
+ "continuous_associations*",
+ "model_table*",
+ "RF_table*"
+ ];
+ const projects = searchPatterns(patterns,projectKeys);
+ return await writeOutput(outPath,projects);
+}
+/**
+ *
+ * @param projectKeyPath
+ * @returns {Promise}
+ */
+const doAll = async function(projectKeyPath,args){
+ const promises = [];
+ const projKeys = await fsPromises.readFile(projectKeyPath,'utf-8');
+ const projectKeys = JSON.parse(projKeys);
+ const LEVELS = args.levels.split(',');
+ detectCombinations(projectKeys)
+
+ promises.push(uniqueProjectsWithSearch(projectKeys,projectKeyPath));
+ promises.push(uniqueProjectsWithCombinationSearch(projectKeys,projectKeyPath));
+ promises.push(uniques(projectKeys,projectKeyPath));
+ promises.push(levels(projectKeys,projectKeyPath,LEVELS));
+ promises.push(features(projectKeys,projectKeyPath));
+ promises.push(searchProjectPatterns(projectKeys,projectKeyPath));
+ promises.push(uniqueProjectKeysWithPertPlates(projectKeys,projectKeyPath));
+
+ const p = await Promise.all(promises);
+ return "done";
+}
+const parser = new ArgumentParser({
+ description: 'Argparse example'
+ });
+
+parser.add_argument('-f', '--compound_key_file', { required: true, help: 'Compound key file' });
+parser.add_argument('-l', '--levels', { default: LEVELS.join(','), help: 'Comma separated list of Levels' });
+
+const args=parser.parse_args()
+console.log(args.levels)
+
+// const allARGS = process.argv;
+// if(allARGS.length != 3){
+// console.log("node projectKeys ");
+// process.exit(1);
+// }
+
+const projectKey = args.compound_key_file;
+
+const p = doAll(projectKey,args);
+
+p.then(function(data){
+ console.log(data);
+ process.exit(0);
+}).catch(function (err) {
+ console.log(err);
+ process.exit(1);
+});
+
+
+
+
diff --git a/convert-projectkeys-csv-to-json/package-lock.json b/convert-projectkeys-csv-to-json/package-lock.json
new file mode 100644
index 0000000..5e86fc9
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/package-lock.json
@@ -0,0 +1,39 @@
+{
+ "name": "convert-projectkeys-csv-to-json",
+ "version": "1.0.0",
+ "lockfileVersion": 2,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "convert-projectkeys-csv-to-json",
+ "version": "1.0.0",
+ "license": "ISC",
+ "dependencies": {
+ "argparse": "^2.0.1",
+ "underscore": "^1.13.1"
+ }
+ },
+ "node_modules/argparse": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
+ "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q=="
+ },
+ "node_modules/underscore": {
+ "version": "1.13.1",
+ "resolved": "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
+ "integrity": "sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g=="
+ }
+ },
+ "dependencies": {
+ "argparse": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
+ "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q=="
+ },
+ "underscore": {
+ "version": "1.13.1",
+ "resolved": "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
+ "integrity": "sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g=="
+ }
+ }
+}
diff --git a/convert-projectkeys-csv-to-json/package.json b/convert-projectkeys-csv-to-json/package.json
new file mode 100644
index 0000000..e8c30e0
--- /dev/null
+++ b/convert-projectkeys-csv-to-json/package.json
@@ -0,0 +1,15 @@
+{
+ "name": "convert-projectkeys-csv-to-json",
+ "version": "1.0.0",
+ "description": "This module converts compound keys from csv format to json so that downstream modules can read the JSON file from the file system. This will help get around the 8192 character limit for the container overrides portion of AWS Batch Jobs, since we do not have to pass the full compound keys as environment variables to each module",
+ "main": "projectKeys.js",
+ "scripts": {
+ "test": "echo \"Error: no test specified\" && exit 1"
+ },
+ "author": "",
+ "license": "ISC",
+ "dependencies": {
+ "argparse": "^2.0.1",
+ "underscore": "^1.13.1"
+ }
+}
diff --git a/deal/.dockerignore b/deal/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/deal/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/deal/Dockerfile b/deal/Dockerfile
new file mode 100644
index 0000000..2a3f666
--- /dev/null
+++ b/deal/Dockerfile
@@ -0,0 +1,18 @@
+FROM prismcmap/base-prism:latest
+MAINTAINER Anup Jonchhe
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+RUN apt-get install -y jq
+COPY ./deal_bash.sh /clue/bin/deal
+COPY ./deal.py /clue/bin/deal.py
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+
+RUN cd /cmap && git clone https://github.com/cmap/merino.git
+
+WORKDIR /
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["deal"]
diff --git a/deal/README.md b/deal/README.md
new file mode 100644
index 0000000..391f13c
--- /dev/null
+++ b/deal/README.md
@@ -0,0 +1,67 @@
+# Deal Module - Testing Jenkins
+
+## Deal Module Overview
+
+Distributes files to make build for each project
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/deal) run:
+
+```
+docker pull prismcmap/deal:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `deal_bash.sh` which is a wrapper around `deal.py` for more information using python run
+
+```
+$ python deal.py --help
+usage: deal.py [-h] [--build_path BUILD_PATH] [--only_key ONLY_KEY]
+ [--project PROJECT] [--sig_id_cols SIG_ID_COLS] [--out OUT]
+ [--verbose]
+
+Distributes files to make build for each project
+
+optional arguments:
+ -h, --help show this help message and exit
+ --build_path BUILD_PATH, -b BUILD_PATH
+ Build path (default: None)
+ --only_key ONLY_KEY, -k ONLY_KEY
+ key to extract. Useful if parallelizing, only listed
+ keys will be concatenated (default: None)
+ --project PROJECT, -p PROJECT
+ Project to extract (default: None)
+ --sig_id_cols SIG_ID_COLS, -s SIG_ID_COLS
+ Comma separated list of col names to create sig_ids if
+ not present (default:
+ pert_plate,culture,pert_id,pert_idose,pert_time)
+ --out OUT, -o OUT Output for collated build (default: None)
+ --verbose, -v Whether to print a bunch of output (default: False)
+
+```
+
+### Example usage with Docker
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+
+```
+docker run \
+ -it \
+ -v ~/{PROJECT_DIR}:/data/ \
+ prismcmap/deal:latest \
+ -b /data/build \
+ -o /data/projects
+```
+
+
+### Example usage with Python
+
+Running python requires an environment set up to use merino. See merino repo here: https://github.com/cmap/merino
+
+```
+python deal.py -b {PROJECT_DIR}/build -o {PROJECT_DIR}/projects
+```
diff --git a/deal/build_docker.sh b/deal/build_docker.sh
new file mode 100755
index 0000000..017e5b4
--- /dev/null
+++ b/deal/build_docker.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/deal:testing --rm=true .
diff --git a/deal/deal.py b/deal/deal.py
new file mode 100755
index 0000000..9dcda1a
--- /dev/null
+++ b/deal/deal.py
@@ -0,0 +1,308 @@
+"""
+Distributes files to make build for each project
+"""
+
+import os
+import shutil
+import re
+import sys
+import glob
+import logging
+import argparse
+import pandas as pd
+# from cmapPy.pandasGEXpress.parse import parse
+
+logger = logging.getLogger('deal')
+CTL_TYPES = ['ctl_vehicle']
+build_contents_dict = {
+ 'inst_info': {
+ 'search_pattern': '*_inst_info.txt',
+ 'type': 'metadata',
+ 'format': 'tsv'
+ },
+ 'cell_info': {
+ 'search_pattern': '*_cell_info.txt',
+ 'type': 'metadata',
+ 'format': 'tsv'
+ },
+ 'EPS_QC_TABLE': {
+ 'search_pattern': '*QC_TABLE*.csv',
+ 'type': 'report',
+ 'format': 'csv'
+ },
+ 'QC_TABLE': {
+ 'search_pattern': '*QC_TABLE*.csv',
+ 'type': 'report',
+ 'format': 'csv'
+ },
+ 'LEVEL2_COUNT': {
+ 'search_pattern': '*_LEVEL2_COUNT*.csv',
+ 'type': 'data',
+ 'annotated': False,
+ 'format': 'csv',
+ },
+ 'LEVEL2_MFI': {
+ 'search_pattern': '*_LEVEL2_MFI*.csv',
+ 'type': 'data',
+ 'annotated': False,
+ 'format': 'csv'
+ },
+ 'LEVEL3_LMFI': {
+ 'search_pattern': '*_LEVEL3_LMFI*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+ 'LEVEL3_NORMALIZED_COUNTS': {
+ 'search_pattern': '*_LEVEL3_NORMALIZED_COUNTS*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+ 'LEVEL4_LFC': {
+ 'search_pattern': '*_LEVEL4_LFC_n*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+ 'LEVEL4_LFC_COMBAT': {
+ 'search_pattern': '*_LEVEL4_LFC_COMBAT*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+ 'LEVEL5_LFC': {
+ 'search_pattern': '*_LEVEL5_LFC_n*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+ 'LEVEL5_LFC_COMBAT': {
+ 'search_pattern': '*_LEVEL5_LFC_COMBAT*.csv',
+ 'type': 'data',
+ 'annotated': True,
+ 'format': 'csv'
+ },
+}
+
+
+def build_parser():
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ parser.add_argument('--build_path', '-b', help='Build path', required=True)
+ parser.add_argument('--only_key', '-k',
+ help='key to extract. Useful if parallelizing, only listed keys will be concatenated')
+ parser.add_argument('--project', '-p', help='Project to extract')
+ parser.add_argument('--sig_id_cols', '-s',
+ help='Comma separated list of col names to create sig_ids if not present',
+ default='pert_plate,culture,pert_id,pert_idose,pert_time',
+ )
+ parser.add_argument('--ignore_missing', action="store_true", default=False)
+ parser.add_argument('--out', '-o', help='Output for collated build')
+ parser.add_argument("--verbose", '-v', help="Whether to print a bunch of output", action="store_true",
+ default=False)
+
+ return parser
+
+
+"""
+Make sig_ids based on id_cols.
+"""
+
+
+def make_sig_id(level5_table, id_cols):
+ col_hds = id_cols.split(',')
+ level5_table['sig_id'] = level5_table.apply(lambda row: '_'.join([str(row[col]) for col in col_hds]), axis=1)
+ return level5_table
+
+def get_data_plus_controls(data, project):
+ proj_inst = data.loc[
+ (data['x_project_id'] == project)
+ ]
+
+ proj_plates = proj_inst.pert_plate.unique()
+
+ proj_ctls = data.loc[
+ (data['pert_plate'].isin(proj_plates)) &
+ (data['pert_type'].isin(CTL_TYPES))
+ ]
+
+ return pd.concat([proj_inst, proj_ctls])
+
+"""
+ Extract project data and write to file
+"""
+def slice_and_write_project(data, data_level, project, outpath, args):
+ print(project)
+ proj_dir = os.path.join(outpath, project, 'data')
+ if not os.path.exists(proj_dir):
+ os.makedirs(proj_dir)
+
+ # Add sig_ids if not present
+ if ("LEVEL5" in data_level) & ('sig_id' not in data.columns):
+ logger.debug("Generating sig_ids from sig_id_cols arg: {}".format(args.sig_id_cols))
+ data = make_sig_id(data, args.sig_id_cols)
+
+ if not build_contents_dict[data_level]['annotated']: # Only used for level 2
+ assert len(glob.glob(os.path.join(args.build_path, build_contents_dict['inst_info'][
+ 'search_pattern']))) == 1, "Incorrect number of inst_info files found"
+ inst = pd.read_csv(
+ glob.glob(os.path.join(args.build_path, build_contents_dict['inst_info']['search_pattern']))[0],
+ sep='\t',
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'}
+ )
+
+ proj_inst = get_data_plus_controls(inst, project)
+
+ proj_data = data.loc[
+ data['cid'].isin(proj_inst['profile_id'].unique())
+ ]
+
+ nc = len(proj_data['cid'].unique())
+ nr = len(proj_data['rid'].unique())
+ else:
+ if 'pert_type' in data.columns:
+ proj_data = get_data_plus_controls(data, project)
+ else: # protects for when we dropped pert_type from Level5
+ proj_data = data.loc[
+ data['x_project_id'] == project
+ ]
+
+ col_id = ('profile_id' if 'profile_id' in data.columns else 'sig_id')
+ row_id = 'ccle_name'
+
+ nc = len(proj_data[col_id].unique())
+ nr = len(proj_data[row_id].unique())
+
+ proj_data.to_csv(
+ os.path.join(proj_dir, '{}_{}_n{}x{}.csv'.format(project, data_level, nc, nr)),
+ index=False,
+ )
+
+
+def main(args):
+ build_path = args.build_path
+ outpath = args.out
+
+ if args.only_key and args.project:
+ key = args.only_key
+ project = args.project
+ dl_dict = build_contents_dict[key]
+ proj_dir = os.path.join(outpath, project, 'data')
+ if key == 'inst_info':
+ file_paths = glob.glob(os.path.join(build_path, dl_dict['search_pattern']))
+ if args.ignore_missing and len(file_paths) < 1:
+ return
+ inst = pd.read_csv(file_paths[0],
+ sep='\t',
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'}
+ )
+ inst.loc[inst['x_project_id'] == project].to_csv(
+ os.path.join(proj_dir, '{}_{}.txt'.format(project, key)),
+ index=False,
+ sep='\t'
+ )
+ elif key == 'cell_info':
+ shutil.copy(
+ glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0],
+ os.path.join(proj_dir, '{}_cell_info.txt'.format(project))
+ )
+ elif key == 'EPS_QC_TABLE':
+ shutil.copy(
+ glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0],
+ os.path.join(proj_dir, '{}_QC_TABLE.csv'.format(project))
+ )
+ elif key == 'QC_TABLE':
+ file_paths = glob.glob(os.path.join(build_path, dl_dict['search_pattern']))
+ if args.ignore_missing and len(file_paths) < 1:
+ return
+ inst = pd.read_csv(
+ glob.glob(os.path.join(build_path, build_contents_dict['inst_info']['search_pattern']))[0],
+ sep='\t',
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'},
+ )
+ proj_inst = inst.loc[inst['x_project_id'] == project]
+ preps = proj_inst.prism_replicate.unique()
+
+ qc = pd.read_csv(glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0],
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'}
+ )
+ qc.loc[qc['prism_replicate'].isin(preps)].to_csv(
+ os.path.join(proj_dir, '{}_{}.csv'.format(project, key)),
+ index=False
+ )
+ else:
+ file_paths = glob.glob(os.path.join(build_path, dl_dict['search_pattern']))
+ if args.ignore_missing and len(file_paths) < 1:
+ return
+ data = pd.read_csv(glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0])
+ slice_and_write_project(data, key, project, outpath, args)
+
+ else:
+ assert len(glob.glob(os.path.join(build_path, build_contents_dict['inst_info'][
+ 'search_pattern']))) == 1, "Incorrect number of inst_info files found"
+
+ inst = pd.read_csv(
+ glob.glob(os.path.join(build_path, build_contents_dict['inst_info']['search_pattern']))[0],
+ sep='\t',
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'},
+ )
+ qc = pd.read_csv(
+ glob.glob(os.path.join(build_path, build_contents_dict['QC_TABLE']['search_pattern']))[0],
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'}
+ )
+ for project in inst['x_project_id'].unique():
+ if pd.isna(project):
+ continue
+ proj_dir = os.path.join(outpath, project, 'data')
+
+ if not os.path.exists(os.path.join(outpath, project, 'data')):
+ os.makedirs(os.path.join(outpath, project, 'data'))
+
+ proj_inst = inst.loc[
+ inst['x_project_id'] == project
+ ]
+
+ proj_pp_list = proj_inst['prism_replicate'].unique()
+ proj_qc = qc.loc[
+ qc['prism_replicate'].isin(proj_pp_list)
+ ]
+
+ proj_inst.to_csv(os.path.join(proj_dir, '{}_inst_info.csv'.format(project)),
+ index=False)
+ shutil.copy(
+ glob.glob(os.path.join(build_path, build_contents_dict['cell_info']['search_pattern']))[0],
+ os.path.join(proj_dir, '{}_cell_info.txt'.format(project))
+ )
+ proj_qc.to_csv(os.path.join(proj_dir, '{}_QC_TABLE.csv'.format(project)),
+ index=False)
+
+ data_files = {k: v for k, v in build_contents_dict.items() if v['type'] == 'data'}
+
+ for data_level, dl_dict in data_files.items():
+ print(data_level)
+ assert len(glob.glob(os.path.join(build_path, dl_dict[
+ 'search_pattern']))) == 1, "Incorrect number of files for data level: {}".format(data_level)
+
+ print(glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0])
+ data = pd.read_csv(
+ glob.glob(os.path.join(build_path, dl_dict['search_pattern']))[0],
+ dtype={'pert_dose': 'str', 'pert_idose': 'str'}
+ )
+ for project in inst['x_project_id'].unique():
+ if pd.isna(project):
+ continue
+
+ if dl_dict['annotated']:
+ slice_and_write_project(data, data_level, project, outpath, args)
+ else:
+ slice_and_write_project(data, data_level, project, outpath, args)
+
+
+if __name__ == "__main__":
+ args = build_parser().parse_args(sys.argv[1:])
+
+ level = (logging.DEBUG if args.verbose else logging.INFO)
+ logging.basicConfig(level=level)
+ logger.info("args: {}".format(args))
+
+ main(args)
diff --git a/deal/deal_bash.sh b/deal/deal_bash.sh
new file mode 100755
index 0000000..9b34923
--- /dev/null
+++ b/deal/deal_bash.sh
@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+
+#setup environment
+source activate prism
+cd /cmap/merino/
+python setup.py develop
+
+#return to /
+cd /
+
+print_help () {
+ python /clue/bin/deal.py --help
+}
+
+#optional
+if test $# -lt 1; then
+ print_help
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit 0
+ ;;
+ -f| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -b| --build_path)
+ shift
+ BUILD_PATH=$1
+ ;;
+ -p| --project)
+ shift
+ PROJECT=$1
+ ;;
+ -k|--only_keys)
+ shift
+ KEYS=$1
+ ;;
+ -s|--sig_id_cols)
+ shift
+ SIG_ID_COLS=$1
+ ;;
+ -o|--out)
+ shift
+ OUT_DIR=$1
+ ;;
+ -v| --verbose)
+ shift
+ VERBOSE=true
+ ;;
+ --ignore_missing)
+ shift
+ IGNORE_MISSING=true
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+
+
+##Run Collate
+if [[ -z $BUILD_PATH || -z $OUT_DIR ]]
+then
+ printf "Required arguments missing\n"
+ exit -1
+fi
+
+args=(
+ -b "$BUILD_PATH"
+)
+
+if [[ -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ if [[ ! -z $KEYS ]]
+ then
+ args+=(-k $KEYS)
+ fi
+ if [[ ! -z $PROJECT ]]
+ then
+ args+=(-p $PROJECT)
+ fi
+ if [[ ! -z $SIG_ID_COLS ]]
+ then
+ args+=(-s $SIG_ID_COLS)
+ fi
+ batch=0
+else
+ batch=${AWS_BATCH_JOB_ARRAY_INDEX}
+fi
+
+if [[ ! -z $projects ]]
+then
+ PROJECT=$(cat "${projects}" | jq -r --argjson index ${batch} '.[$index].x_project_id')
+ OUT_DIR="${OUT_DIR}"/"${PROJECT,,}"
+ KEY=$(cat "${projects}" | jq -r --argjson index ${batch} '.[$index].level')
+ args+=(-p "$PROJECT")
+ args+=(-k "$KEY")
+fi
+
+if [ ! -d $OUT_DIR ]
+then
+ mkdir -p $OUT_DIR
+fi
+args+=(-o "$OUT_DIR")
+
+if [[ ! -z $VERBOSE ]]
+then
+ args+=(-v)
+fi
+
+if [[ ! -z $IGNORE_MISSING ]]
+then
+ args+=(--ignore_missing)
+fi
+
+echo python /clue/bin/deal.py "${args[@]}"
+
+python /clue/bin/deal.py "${args[@]}"
+
+exit_code=$?
+conda deactivate
+exit $exit_code
diff --git a/deal/docker_push_tags.sh b/deal/docker_push_tags.sh
new file mode 100755
index 0000000..a070f3a
--- /dev/null
+++ b/deal/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/deal:testing
+
+
+# trigger
\ No newline at end of file
diff --git a/depmap_versions/.dockerignore b/depmap_versions/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/depmap_versions/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/depmap_versions/Dockerfile b/depmap_versions/Dockerfile
new file mode 100644
index 0000000..a1a0392
--- /dev/null
+++ b/depmap_versions/Dockerfile
@@ -0,0 +1,18 @@
+FROM cmap/base-clue-mts
+MAINTAINER Andrew Boghossian
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./src/depmap_functions.R /src/depmap_functions.R
+COPY ./make_matrices.R /make_matrices.R
+COPY ./aws_batch.sh /clue/bin/aws_batch
+
+RUN ["chmod","+x", "/make_matrices.R"]
+RUN ["chmod","+x", "/src/depmap_functions.R"]
+
+WORKDIR /
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/depmap_versions/README.md b/depmap_versions/README.md
new file mode 100644
index 0000000..bac29ef
--- /dev/null
+++ b/depmap_versions/README.md
@@ -0,0 +1,52 @@
+# DepMap Versions Module
+
+This module makes wide versions (matrices) of log-fold change, AUC, and IC50 that are compatible with upload to the [DepMap portal](https://depmap.org/portal/interactive/).
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/depmap_versions) run:
+
+```
+docker pull prismcmap/depmap_versions:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `aws_batch.sh` which is a wrapper around `make_matrices.R` for more information using R run
+
+```
+Rscript make_matrices.R --help
+usage: make_matrices.R [-h] [-p PROJECT_DIR] [-o OUT] [-n NAME]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -p PROJECT_DIR, --project_dir PROJECT_DIR
+ Project directory
+ -o OUT, --out OUT Output directory
+ -n NAME, --name NAME Build name. Default is none
+```
+
+### Example usage with R
+
+R execution requires installing the correct R packages which are outlined in the `docker_base` module.
+
+```
+Rscript make_matrices.R -p ~/Desktop/clue_data/project -o ~/Desktop/mts_results/project -n PMTS001
+```
+
+### Example usage with Docker
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+```
+docker run \
+ -it \
+ -v ~/Desktop/clue_data/project:/in_data \
+ -v ~/Desktop/mts_results/project:/out_data \
+ prismcmap/depmap_versions:latest \
+ -p /in_data \
+ -o /out_data \
+ -n PMTS001
+```
diff --git a/depmap_versions/aws_batch.sh b/depmap_versions/aws_batch.sh
new file mode 100644
index 0000000..868533a
--- /dev/null
+++ b/depmap_versions/aws_batch.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# read in flagged arguments
+while test $# -gt 0; do
+ case "$1" in
+ -f| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -p| --project_dir)
+ shift
+ project_dir=$1
+ ;;
+ -n| --build_name)
+ shift
+ build_name=$1
+ ;;
+ -o| --output_dir)
+ shift
+ output_dir=$1
+ ;;
+ -lp| --lfc_pattern)
+ shift
+ lfc_pattern=$1
+ ;;
+ -lc| --lfc_column)
+ shift
+ lfc_column=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+batch_index=0
+sanitized_pert_id=""
+if [[ ! -z $projects ]]
+then
+ if [[ ! -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+ fi
+ project=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ project_dir="${project_dir}"/"${project,,}"/"${project^^}"/data/
+ build_name="${project^^}"
+ output_dir="${project_dir}"
+fi
+
+echo "${project_dir}" "${output_dir}" "${build_name}"
+
+args=(
+ -p "${project_dir}"
+ -o "${output_dir}"
+ -n "${build_name}"
+)
+
+
+if [[ ! -z $lfc_pattern ]]
+then
+ args+=(-lp "${lfc_pattern}")
+fi
+
+if [[ ! -z $lfc_column ]]
+then
+ args+=(-lc "${lfc_column}")
+fi
+
+if [[ "$sanitized_pert_id" == "DMSO" ]]
+then
+ echo "Skipping DMSO"
+else
+ Rscript /make_matrices.R "${args[@]}"
+fi
+
+
+exit_code=$?
+
+echo "$exit_code"
+exit $exit_code
diff --git a/depmap_versions/build_docker.sh b/depmap_versions/build_docker.sh
new file mode 100755
index 0000000..001df5b
--- /dev/null
+++ b/depmap_versions/build_docker.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/depmap_versions:develop --rm=true .
diff --git a/depmap_versions/docker_push_tags.sh b/depmap_versions/docker_push_tags.sh
new file mode 100755
index 0000000..1127bdc
--- /dev/null
+++ b/depmap_versions/docker_push_tags.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+docker push prismcmap/depmap_versions:develop
+
+# triggers
\ No newline at end of file
diff --git a/depmap_versions/docker_run.sh b/depmap_versions/docker_run.sh
new file mode 100755
index 0000000..f34a7f3
--- /dev/null
+++ b/depmap_versions/docker_run.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+docker run --rm \
+--name bar \
+-v ~/WebstormProjects/PMTS018/projects/:/data \
+-it prismcmap/landing \
+-d /data/mts018_dmc_loxo/MTS018_DMC_LOXO \
+-o /data/mts018_dmc_loxo \
+-p MTS018_DMC_LOXO
diff --git a/depmap_versions/make_matrices.R b/depmap_versions/make_matrices.R
new file mode 100644
index 0000000..1550c71
--- /dev/null
+++ b/depmap_versions/make_matrices.R
@@ -0,0 +1,63 @@
+# Script to make DepMap compatible matrices
+
+# import necessary libraries and functions using MTS_functions.R
+suppressMessages(source("./src/depmap_functions.R"))
+
+#---- Read arguments ----
+
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-p", "--project_dir", default="", help="Project directory")
+parser$add_argument("-o", "--out", default="", help="Output directory")
+parser$add_argument("-n", "--name", default="", help = "Build name. Default is none")
+parser$add_argument("-lp", "--lfc_pattern", default="LEVEL5_LFC_COMBAT", help = "LFC file search pattern")
+parser$add_argument("-lc", "--lfc_column", default="LFC_cb", help = "LFC column to use - LFC or lfc_cb")
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+
+proj_dir <- args$project_dir
+out_dir <- args$out
+build_name <- args$name
+lfc_pattern <- args$lfc_pattern
+lfc_column <- args$lfc_column
+
+# get paths to LFC and DRC
+lfc_path <- list.files(proj_dir, pattern = lfc_pattern, full.names = T)
+stopifnot(length(lfc_path) == 1) # need LFC
+drc_path <- list.files(proj_dir, pattern = "DRC_TABLE", full.names = T)
+
+print("line 26")
+# read in LFC
+lfc_tab <- data.table::fread(lfc_path) %>%
+ dplyr::filter(!is.na(ccle_name), pool_id != "CTLBC",
+ str_detect(ccle_name, "prism invariant", negate = T))
+print("line 31")
+# pivot
+lfc_mat <- reshape2::acast(lfc_tab, ccle_name ~ pert_iname + pert_idose,
+ value.var = lfc_column,
+ fun.aggregate = function(x) mean(x, na.rm = TRUE))
+# write LFC
+write.csv(lfc_mat, paste0(out_dir, "/", build_name, "_LFC_MATRIX.csv"))
+
+print("line 39")
+# do the same for DRC if it exists (make AUC and IC50 matrix)
+if (length(drc_path) == 1) {
+ drc_tab <- data.table::fread(drc_path) %>%
+ dplyr::filter(!is.na(ccle_name),
+ str_detect(ccle_name, "prism invariant", negate = T)) %>%
+ dplyr::mutate(condition = varied_iname)
+ print("line 46")
+ if ("added_compounds" %in% colnames(drc_tab)) {
+ drc_tab$condition <- paste(drc_tab$condition, drc_tab$added_compounds, drc_tab$added_doses, sep = "_")
+ }
+ auc_mat <- reshape2::acast(drc_tab, ccle_name ~ condition,
+ value.var = "auc",
+ fun.aggregate = function(x) mean(x, na.rm = TRUE))
+ ic50_mat <- reshape2::acast(drc_tab, ccle_name ~ condition,
+ value.var = "log2.ic50",
+ fun.aggregate = function(x) mean(x, na.rm = TRUE))
+
+ write.csv(auc_mat, paste0(out_dir, "/", build_name, "_AUC_MATRIX.csv"))
+ write.csv(ic50_mat, paste0(out_dir, "/", build_name, "_IC50_MATRIX.csv"))
+}
diff --git a/depmap_versions/src/depmap_functions.R b/depmap_versions/src/depmap_functions.R
new file mode 100644
index 0000000..aa00a32
--- /dev/null
+++ b/depmap_versions/src/depmap_functions.R
@@ -0,0 +1,8 @@
+#---- Load Packages ----
+library(tidyverse)
+library(magrittr)
+library(data.table)
+library(readr)
+library(stats)
+library(reshape2)
+library(argparse)
diff --git a/docker_base/.dockerignore b/docker_base/.dockerignore
new file mode 100644
index 0000000..d7d074e
--- /dev/null
+++ b/docker_base/.dockerignore
@@ -0,0 +1,2 @@
+*.sh
+README.md
diff --git a/docker_base/Dockerfile b/docker_base/Dockerfile
new file mode 100644
index 0000000..5159dee
--- /dev/null
+++ b/docker_base/Dockerfile
@@ -0,0 +1,22 @@
+FROM rocker/r-ver:4.0.2
+MAINTAINER Andrew Boghossian
+LABEL base.mts.pipeline.clue.io.version="0.0.1"
+LABEL base.mts.pipeline.clue.io.vendor="PRISM"
+
+RUN apt-get update -qq && \
+ apt-get -y upgrade
+RUN mkdir -p /src && apt-get -y --no-install-recommends install \
+ libssl-dev \
+ libxml2-dev \
+ libcairo2-dev \
+ libsqlite-dev \
+ libmariadbd-dev \
+ libmariadbclient-dev \
+ libpq-dev \
+ libssh2-1-dev \
+ libhdf5-dev \
+ libcurl4-openssl-dev \
+ jq
+
+COPY ./install_packages.R /src/install_packages.R
+RUN mkdir -p /clue/bin && Rscript /src/install_packages.R
diff --git a/docker_base/README.md b/docker_base/README.md
new file mode 100644
index 0000000..54781e0
--- /dev/null
+++ b/docker_base/README.md
@@ -0,0 +1,22 @@
+# Docker Base
+
+For making the base image for clue-mts docker. Installs necessary R packages to run the MTS pipeline.
+
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/cmap/base-clue-mts) run:
+
+```
+docker pull cmap/base-clue-mts:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Installing packages locally
+
+If you have R installed and would like to install the packages to your local machine run
+
+```
+Rscript install_packages.R
+```
diff --git a/docker_base/build_docker.sh b/docker_base/build_docker.sh
new file mode 100755
index 0000000..7efec21
--- /dev/null
+++ b/docker_base/build_docker.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t cmap/base-clue-mts:latest -t cmap/base-clue-mts:v0.0.1 --rm=true .
diff --git a/docker_base/docker_push_tags.sh b/docker_base/docker_push_tags.sh
new file mode 100755
index 0000000..a202f98
--- /dev/null
+++ b/docker_base/docker_push_tags.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+docker push cmap/base-clue-mts:v0.0.1
+docker push cmap/base-clue-mts:latest
+
+# trigger
\ No newline at end of file
diff --git a/src/install_packages.R b/docker_base/install_packages.R
similarity index 51%
rename from src/install_packages.R
rename to docker_base/install_packages.R
index e714ce4..38793aa 100644
--- a/src/install_packages.R
+++ b/docker_base/install_packages.R
@@ -5,11 +5,19 @@ install.packages("hdf5r")
install.packages("reshape2")
install.packages("readr")
install.packages("magrittr")
-install.packages("dr4pl")
+install.packages("dr4pl") ## for DRC curve fit
+install.packages("drc") ## for DRC curve fit
install.packages("data.table")
install.packages("scam")
+install.packages("argparse")
+install.packages("splitstackshape")
+install.packages("R.utils")
install.packages("BiocManager")
BiocManager::install()
BiocManager::install("sva")
+BiocManager::install("limma", ask = FALSE)
install.packages("PRROC")
install.packages("gmodels")
+install.packages("R.utils")
+install.packages("devtools")
+devtools::install_github("broadinstitute/cdsr_models", dependencies=TRUE, force=TRUE)
diff --git a/docker_wrapper.sh b/docker_wrapper.sh
deleted file mode 100644
index 486e0dd..0000000
--- a/docker_wrapper.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/bash
-
-# read in flagged arguments
-while getopts ":i:o:p:" arg; do
- case $arg in
- i) # specify input folder
- data_dir=${OPTARG}
- ;;
- o) # specifcy output folder
- output_dir=${OPTARG}
- ;;
- p) # specify project name
- project=${OPTARG}
- ;;
- esac
-done
-
-# run docker container with arguments
-docker run -it \
- -v $data_dir:/data \
- -v $output_dir:/results \
- aboghoss/clue-mts "data" "results" "$project"
diff --git a/dockerized_mts.Rproj b/dockerized_mts.Rproj
deleted file mode 100644
index 8e3c2eb..0000000
--- a/dockerized_mts.Rproj
+++ /dev/null
@@ -1,13 +0,0 @@
-Version: 1.0
-
-RestoreWorkspace: Default
-SaveWorkspace: Default
-AlwaysSaveHistory: Default
-
-EnableCodeIndexing: Yes
-UseSpacesForTab: Yes
-NumSpacesForTab: 2
-Encoding: UTF-8
-
-RnwWeave: Sweave
-LaTeX: pdfLaTeX
diff --git a/drc/.dockerignore b/drc/.dockerignore
new file mode 100644
index 0000000..850bd8c
--- /dev/null
+++ b/drc/.dockerignore
@@ -0,0 +1,5 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
+enter_container.sh
diff --git a/drc/Dockerfile b/drc/Dockerfile
new file mode 100644
index 0000000..0d1cb55
--- /dev/null
+++ b/drc/Dockerfile
@@ -0,0 +1,19 @@
+FROM cmap/base-clue-mts
+MAINTAINER Andrew Boghossian
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+COPY ./install_packages.R /src/install_packages.R
+RUN mkdir -p /clue/bin && Rscript /src/install_packages.R
+
+COPY ./src/drc_functions.R /src/drc_functions.R
+COPY ./drc_compound.R /drc_compound.R
+COPY ./aws_batch.sh /clue/bin/aws_batch
+RUN chmod -R +x /*.R
+
+WORKDIR /
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/drc/README.md b/drc/README.md
new file mode 100644
index 0000000..febc838
--- /dev/null
+++ b/drc/README.md
@@ -0,0 +1,63 @@
+# DRC Module
+
+This module fits dose-response curves to the viability values for each cell line. It requires the output of the LFC module.
+
+## Dose-response overview
+
+For each cell line passing QC, we attempt to fit a 4-parameter log logistic dose-response curve and for each curve we compute summary statistics including AUC and IC50. An IC50 is reported only if the fitted curve crosses 50% viability within the dose range of the experiment. If the curve fit does not succeed, fit parameters are omitted, and only the Riemann AUC is provided.
+
+The fit is performed with the following restrictions:
+- For single-agent studies, We support the predicted viability to decrease with increasing dose by constraining the slope parameter (s> 1e-5). For combination studies, we relax this slope constraint since the two agents could be antagonistic.
+
+- For single-agent studies, we constrain the upper limit of the fit to be between 0.8 and 1.01. For combination studies, this constraint is relaxed as the viability at the anchor dose can be lower, and the upper limit of the fit is between 0 and 1.
+
+- We constrain the lower limit of the fit to be between 0 and 1.
+
+Since the nonlinear optimization underlying the curve fit can return a local optimum, we first fit the data using a number of different optimization methods and initial conditions implemented in the [drc](https://cran.r-project.org/web/packages/drc/drc.pdf) and [dr4pl](https://cran.r-project.org/web/packages/dr4pl/dr4pl.pdf) packages, and then report the fit with the smallest mean squared error (MSE). The dr4pl package provides a robust fit in some scenarios where the naive fit fails to converge. The functions used to calculate the fit values are located in [`src/drc_functions.R`](./src/drc_functions.R).
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/drc-module) run:
+
+```
+docker pull prismcmap/drc-module:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `aws_batch.sh` which is a wrapper around `drc_compound.R` for more information using R run
+
+```
+Rscript drc_compound.R --help
+usage: drc_compound.R [-h] [-i INPUT_DIR] [-o OUT]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -i INPUT_DIR, --input_dir INPUT_DIR
+ Input directory with one level 4 LFC file
+ -o OUT, --out OUT Output directory
+```
+
+### Example usage with R
+
+R execution requires installing the correct R packages which are outlined in the `docker_base` module.
+
+```
+Rscript drc_compound.R -i ~/Desktop/clue_data/project/compound_1 -o ~/Desktop/mts_results/project/compound_1
+```
+
+### Example usage with Docker
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+```
+docker run \
+ -it \
+ -v ~/Desktop/clue_data/project/compound_1:/in_data \
+ -v ~/Desktop/mts_results/project/compound_1:/out_data \
+ prismcmap/drc-module:latest \
+ -i /in_data \
+ -o /out_data
+```
diff --git a/drc/aws_batch.sh b/drc/aws_batch.sh
new file mode 100644
index 0000000..4ae7b54
--- /dev/null
+++ b/drc/aws_batch.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+# read in flagged arguments
+while getopts ":i:o:f:s:" arg; do
+ case $arg in
+ i) # specify input folder
+ data_dir=${OPTARG};;
+ o) # specifcy output folder
+ output_dir=${OPTARG};;
+ f)
+ projects=${OPTARG};;
+ s)
+ screen_type=${OPTARG};;
+ esac
+done
+
+out="${output_dir}"
+data="${data_dir}"
+screen="${screen_type}"
+echo "${data}" "${out}"
+
+batch_index=0
+
+
+if [[ ! -z $projects ]]
+then
+ if [[ ! -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]
+ then
+ batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+ fi
+ pert_id=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_id')
+ project=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ plate=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_plate')
+ cleaned_pert_id=$(echo "${pert_id//|/$'_'}")
+ sanitized_pert_id="${cleaned_pert_id^^}"
+ data="${data_dir}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"
+ out="${output_dir}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"
+fi
+
+args=(
+ -i "${data}"
+ -o "${out}"
+)
+
+# if [[ ! -z $screen ]]
+# then
+# args+=(-s "${screen}")
+# fi
+
+echo "${data}" "${out}"
+if [[ "$pert_id" == "DMSO" ]]
+then
+ echo "Skipping DMSO"
+else
+ echo Rscript /drc_compound.R "${args[@]}"
+ Rscript /drc_compound.R "${args[@]}"
+fi
+
+exit_code=$?
+echo "$exit_code"
+exit $exit_code
diff --git a/drc/build_docker.sh b/drc/build_docker.sh
new file mode 100755
index 0000000..181de0d
--- /dev/null
+++ b/drc/build_docker.sh
@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+docker build --platform linux/amd64 -t prismcmap/drc:testing --rm=true .
diff --git a/drc/docker_push_tags.sh b/drc/docker_push_tags.sh
new file mode 100755
index 0000000..9f2530d
--- /dev/null
+++ b/drc/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+#docker push prismcmap/drc-module:v0.0.1
+docker push prismcmap/drc:production
+
+# trigger
\ No newline at end of file
diff --git a/drc/docker_run.sh b/drc/docker_run.sh
new file mode 100755
index 0000000..11b41d7
--- /dev/null
+++ b/drc/docker_run.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+#docker run --rm \
+#--name bar2 \
+#-v /Users/jasiedu/WebstormProjects/dockerized_mts:/cmap/macchiato \
+#-it prismcmap/drc-module \
+#-i /cmap/macchiato/foo \
+#-o /cmap/macchiato/foo
+
+docker run -it \
+ -v /foo/BRD-K00003576:/in_data \
+ -v /foo/BRD-K00003576:/out_data \
+ prismcmap/drc:robust_DRC \
+ -i /in_data \
+ -o /out_data
+
+#docker run -it -v /Users/anup/work/1_Projects/wkdir/drc_robust/MTS022_VALIDATION_COMPOUNDS_INTERNAL/PMTS061/BRD-K00003576:/in_data --entrypoint /bin/bash 66bd8fb4c4de
\ No newline at end of file
diff --git a/drc/drc_compound.R b/drc/drc_compound.R
new file mode 100644
index 0000000..c165005
--- /dev/null
+++ b/drc/drc_compound.R
@@ -0,0 +1,283 @@
+# Script to fit Dose-Response Curve to LFC data (using all replicates)
+
+# import necessary libraries and functions using MTS_functions.R
+suppressMessages(source("./src/drc_functions.R"))
+
+#---- Read arguments ----
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-i", "--input_dir", default="", help="Input directory with one level 4 LFC file")
+parser$add_argument("-o", "--out", default="", help="Output directory")
+
+## For single agent screens, like MTS, the UpperLimit (UL) of fit is close to 1
+## and fit is constrained to be always decreasing as we assume the compound is always toxic
+## For combination screens, like CPS, the UpperLimit (UL) is b/w 0 and 1 and fit can be both increasing or decreasing.
+## Increasing models antagonistic effects of the second compound.
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+lfc_dir <- args$input_dir
+out_dir <- args$out
+
+fit_type <- "single_agent" ## default screen type is MTS
+
+
+# find level 4 file and return error if none or more than one
+lfc_files <- list.files(lfc_dir, pattern=("*LEVEL4_LFC_.*\\.csv$"), full.names=T)
+if (length(lfc_files) == 1) {
+ lfc_file <- lfc_files[[1]]
+} else if (length(lfc_files) == 2) {
+ combat_file <- grep("COMBAT", lfc_files, value=TRUE)
+ if (length(combat_file) == 1) {
+ lfc_file <- combat_file[[1]]
+ } else {
+ stop("There are 2 LFC files, but no COMBAT file. Please try again with 1 LFC file.",
+ call. = FALSE)
+ }
+} else {
+ stop(paste("There are", length(lfc_files), "LFC files in the supplied directory. Please try again with 1."),
+ call. = FALSE)
+}
+
+#---- Load the data ----
+print("Loading data and pre-processing")
+
+LFC_TABLE <- data.table::fread(lfc_file)
+
+if ("LFC_cb" %in% colnames(LFC_TABLE)) {
+ LFC_column <- "LFC_cb"
+} else {
+ LFC_column <- "LFC"
+}
+
+#---- Pre-processing ----
+# table with each compound cell line combo and number of doses
+compound_table <- LFC_TABLE %>%
+ dplyr::select(pert_iname, pert_id, pert_dose) %>%
+ dplyr::distinct() %>%
+ splitstackshape::cSplit(splitCols = c("pert_dose", "pert_iname", "pert_id"),
+ sep = "|", fixed = T,
+ direction = "wide", drop = T) %>%
+ dplyr::group_by(across(-c(colnames(.)[str_detect(colnames(.), pattern = "pert_dose")]))) %>%
+ summarise_all(function(x) n_distinct(x, na.rm = T)) %>%
+ dplyr::ungroup()
+
+
+# check which compounds were run at 4 or more doses (to fit curves)
+dosed_compounds <- compound_table %>%
+ tidyr::pivot_longer(cols = contains("pert_iname"), names_to = c("foo", "bar", "index"), names_sep = "_", values_to = "pert_iname")
+dosed_compounds$n_doses <- dosed_compounds[1, paste0("pert_dose_", dosed_compounds$index)] %>% t() %>% as.numeric()
+dosed_compounds$pert_id <- dosed_compounds[1, paste0("pert_id_", dosed_compounds$index)] %>% t() %>% as.character()
+dosed_compounds %<>%
+ dplyr::select(pert_iname, pert_id, n_doses, index) %>%
+ dplyr::filter(n_doses >= 4)
+
+
+# if none exit
+if (nrow(dosed_compounds) < 1) {
+ message("Not enough dose points to fit curves for compound(s) in table")
+ quit(save = "no")
+}
+
+# distinct cell line/dose combinations
+DRC_TABLE_cb <- LFC_TABLE %>%
+ dplyr::distinct(pert_iname, pert_time, pert_dose, pert_iname, pert_id, ccle_name, culture, pool_id, pert_plate) %>%
+ splitstackshape::cSplit(splitCols = c("pert_iname", "pert_id", "pert_dose"),
+ sep = "|", fixed = T,
+ direction = "wide", drop = T, type.convert = T)
+# widened version for joining
+LFC_TABLE.split <- LFC_TABLE %>%
+ splitstackshape::cSplit(splitCols = c("pert_iname", "pert_id", "pert_dose"),
+ sep = "|", fixed = T,
+ direction = "wide", drop = F, type.convert = T)
+# if this was a combination study, we need to change fit limits accordingly
+if (any(str_detect(colnames(LFC_TABLE.split), "pert_id_2"))) { ## a second pert existed.
+ fit_type <- "combination"
+ print("Screen type was=CPS, fit type was combination")
+}
+print (paste("fit type_was= ", fit_type))
+
+#---- Compute dose-response parameters ----
+DRC <- list() # stores dose response results
+
+# for each compound run at 4+ doses
+for (i in 1:nrow(dosed_compounds)){
+ print (paste("compound:", dosed_compounds[i,]))
+ comp <- dosed_compounds[i, ]
+ dose_var <- paste0("pert_dose_", comp$index) # index of compound
+
+ # find all cell line drug combos for that compound with 4+ doses
+ df <- DRC_TABLE_cb %>%
+ dplyr::group_by(across(!contains(comp$index))) %>%
+ dplyr::summarise(n = n(), .groups = "drop") %>%
+ dplyr::filter(n >= 4)
+
+ # skip if no cell lines with 4+ doses
+ if (nrow(df) < 1) {
+ DRC[[i]] <- tibble()
+ next
+ }
+
+ sub_DRC <- list() # stores dose response results
+
+ # for each cell line
+ for (j in 1:nrow(df)) {
+
+ # get LFC data
+ d <- df[j, ] %>% dplyr::inner_join(LFC_TABLE.split) %>% suppressMessages()
+ d %<>% dplyr::filter(is.finite(.[[LFC_column]]),is.finite(.[[dose_var]]) ) # drop any infinite LFC values or undefined doses
+
+ d$FC <- 2^d[[LFC_column]] # get fold-change values.
+
+ # fit curve. if MTS UL is close to 1 and slope is always decreasing.
+ # if CPS, UL can be further from 1 since anchor dose can be toxic and slope can increase
+ if (fit_type=="single_agent"){
+ fit_result.df <- get_best_fit(d, dose_var,
+ UL_low=0.8, UL_up=1.01, slope_decreasing=TRUE)
+ }else if(fit_type=="combination"){
+ fit_result.df <- get_best_fit(d, dose_var,
+ UL_low=0.0, UL_up=1.01, slope_decreasing=FALSE)
+ }else{ ## if screen type has not been defined before, then stop, we need to add it here.
+ print("undefined fit type")
+ stop("Error: undefined fit type")
+ }
+
+
+
+ # get results if fit
+ if (fit_result.df$successful_fit) {
+
+ # tibble of curve parameters
+ x <- tibble(min_dose = min(d[[dose_var]]),
+ max_dose = max(d[[dose_var]]),
+ upper_limit = fit_result.df$Upper_Limit,
+ ec50 = fit_result.df$Inflection,
+ slope = -fit_result.df$Slope, ##### sign of slope is made negative to remain compatible with sign convention in report generation module
+ lower_limit = fit_result.df$Lower_Limit,
+ convergence = fit_result.df$successful_fit) %>%
+ dplyr::mutate(auc = compute_auc(lower_limit, upper_limit, ec50, slope,
+ min_dose, max_dose),
+ log2.ic50 = compute_log_ic50(lower_limit, upper_limit,
+ ec50, slope,
+ min_dose, max_dose),
+ auc_riemann=fit_result.df$auc_riemann,
+ mse = fit_result.df$MSE,
+ R2 = fit_result.df$frac_var_explained, ### updated. old R^2 values used the variance of LFC and not FC in denominator
+ best_fit_name = fit_result.df$fit_name,
+ varied_iname = comp$pert_iname,
+ varied_id = comp$pert_id,
+ ccle_name = df[j,]$ccle_name,
+ culture = df[j,]$culture,
+ pool_id = df[j,]$pool_id,
+ pert_time = df[j,]$pert_time,
+ pert_plate = df[j,]$pert_plate,
+ fit_type= fit_type)
+
+ # if this was a combination track other compounds added
+ if (any(str_detect(colnames(df), "pert_id_"))) {
+ added_comp_table <- df[j, ] %>%
+ tidyr::unite(col = added_compounds, starts_with("pert_iname_"), sep = "|") %>%
+ tidyr::unite(col = added_doses, starts_with("pert_dose_"), sep = "|") %>%
+ tidyr::unite(col = added_ids, starts_with("pert_id_"), sep = "|") %>%
+ dplyr::select(-n)
+ x %<>% dplyr::left_join(added_comp_table)%>% suppressMessages()
+ }
+ sub_DRC[[j]] <- x
+ } else { # fit is unsuccessful and so all we can really provide is the riemann auc but we add the other columns as well for completeness
+ x <- tibble(min_dose = min(d[[dose_var]]),
+ max_dose = max(d[[dose_var]]),
+ upper_limit = as.numeric(NA),
+ ec50 = as.numeric(NA),
+ slope = as.numeric(NA),
+ lower_limit =as.numeric(NA),
+ convergence = fit_result.df$successful_fit) %>%
+ dplyr::mutate(auc = as.numeric(NA),
+ log2.ic50 = as.numeric(NA),
+ auc_riemann=fit_result.df$auc_riemann,
+ mse = as.numeric(NA),
+ R2 = as.numeric(NA),
+ best_fit_name = NA,
+ varied_iname = comp$pert_iname,
+ varied_id = comp$pert_id,
+ ccle_name = df[j,]$ccle_name,
+ culture = df[j,]$culture,
+ pool_id = df[j,]$pool_id,
+ pert_time = df[j,]$pert_time,
+ pert_plate = df[j,]$pert_plate,
+ fit_type= fit_type)
+ # if this was a combination track other compounds added
+ if (any(str_detect(colnames(df), "pert_id_"))) {
+ added_comp_table <- df[j, ] %>%
+ tidyr::unite(col = added_compounds, starts_with("pert_iname_"), sep = "|") %>%
+ tidyr::unite(col = added_doses, starts_with("pert_dose_"), sep = "|") %>%
+ tidyr::unite(col = added_ids, starts_with("pert_id_"), sep = "|") %>%
+ dplyr::select(-n)
+ x %<>% dplyr::left_join(added_comp_table)%>% suppressMessages()
+ }
+ sub_DRC[[j]] <- x
+ # sub_DRC[[j]] <- tibble(min_dose = min(d[[dose_var]]),######
+ # max_dose = max(d[[dose_var]]),
+ # upper_limit = as.numeric(NA),
+ # ec50 = as.numeric(NA),
+ # slope = as.numeric(NA), ##### sign of slope is made negative to remain compatible with sign convention in report generation module
+ # lower_limit = as.numeric(NA),
+ # convergence = FALSE) %>%
+ # dplyr::mutate(auc = as.numeric(NA),
+ # log2.ic50 = as.numeric(NA),
+ # auc_riemann=fit_result.df$auc_riemann,
+ # mse = ifelse("mse" %in% colnames(fit_result.df), fit_result.df$MSE, as.numeric(NA) ),
+ # R2 = ifelse("frac_var_explained" %in% colnames(fit_result.df), fit_result.df$frac_var_explained, as.numeric(NA) ), ### updated. old R^2 values used the variance of LFC and not FC in denominator
+ # best_fit_name = NA,
+ # varied_iname = comp$pert_iname,
+ # varied_id = comp$pert_id,
+ # ccle_name = df[j,]$ccle_name,
+ # culture = df[j,]$culture,
+ # pool_id = df[j,]$pool_id,
+ # pert_time = df[j,]$pert_time,
+ # pert_plate = df[j,]$pert_plate)
+ }
+ if (is.na(fit_result.df$successful_fit)){
+ x <- tibble(min_dose = min(d[[dose_var]]),######
+ max_dose = max(d[[dose_var]]),
+ upper_limit = as.numeric(NA),
+ ec50 = as.numeric(NA),
+ slope = as.numeric(NA), ##### sign of slope is made negative to remain compatible with sign convention in report generation module
+ lower_limit = as.numeric(NA),
+ convergence = FALSE) %>%
+ dplyr::mutate(auc = as.numeric(NA),
+ log2.ic50 = as.numeric(NA),
+ auc_riemann=as.numeric(NA),
+ mse = as.numeric(NA),
+ R2 = as.numeric(NA), ### updated. old R^2 values used the variance of LFC and not FC in denominator
+ best_fit_name = NA,
+ varied_iname = comp$pert_iname,
+ varied_id = comp$pert_id,
+ ccle_name = df[j,]$ccle_name,
+ culture = df[j,]$culture,
+ pool_id = df[j,]$pool_id,
+ pert_time = df[j,]$pert_time,
+ pert_plate = df[j,]$pert_plate,
+ fit_type= fit_type)
+ if (any(str_detect(colnames(df), "pert_id_"))) {
+ added_comp_table <- df[j, ] %>%
+ tidyr::unite(col = added_compounds, starts_with("pert_iname_"), sep = "|") %>%
+ tidyr::unite(col = added_doses, starts_with("pert_dose_"), sep = "|") %>%
+ tidyr::unite(col = added_ids, starts_with("pert_id_"), sep = "|") %>%
+ dplyr::select(-n)
+ x %<>% dplyr::left_join(added_comp_table)%>% suppressMessages()
+ }
+ sub_DRC[[j]] <- x
+ }
+
+ }
+
+ sub_DRC %<>% dplyr::bind_rows() # combine results
+ DRC[[i]] <- sub_DRC
+}
+DRC %<>% dplyr::bind_rows() # combine results
+
+
+# #---- Write to .csv ----
+if(nrow(DRC) > 0) {
+ write.csv(DRC, paste0(out_dir, "/DRC_TABLE.csv"), row.names=FALSE)
+}
\ No newline at end of file
diff --git a/drc/install_packages.R b/drc/install_packages.R
new file mode 100644
index 0000000..afd656f
--- /dev/null
+++ b/drc/install_packages.R
@@ -0,0 +1 @@
+install.packages("drc") ## for DRC curve fit
diff --git a/drc/src/drc_functions.R b/drc/src/drc_functions.R
new file mode 100644
index 0000000..a7eec48
--- /dev/null
+++ b/drc/src/drc_functions.R
@@ -0,0 +1,490 @@
+# Helper functions for the initial processing step of the MTS pipeline
+
+#---- Load Packages ----
+library(tidyverse)
+library(magrittr)
+library(data.table)
+library(dr4pl)
+library(drc)
+library(readr)
+library(stats)
+library(reshape2)
+library(argparse)
+library(splitstackshape)
+
+## NOTE: The sign conventions for the slope parameter in the drc and dr4pl packages are opposite to each other. i.e slope_drc=-slope_dr4pl
+## The data is eventually saved by the drc_compound.R script and passed to future modules with the sign convention of the drc package
+## The functions "compute_log_ic50" and "compute_auc" use a sign convention for the slope parameter consistent with drc package and are called from outside this script.
+## The function "compute_MSE_MAD", called from within this script using the sign convention of the dr4pl package.
+
+#---- Dose-Response Parameters ----
+# l or LL is the lower limit
+# u or UL is the upper limit
+# Slope or h is the slope
+# EC50 or Inflection is the inflection point, which is also called EC50
+
+# area under curve given dose-response parameters
+compute_auc <- function(l, u, ec50, h, md, MD) {
+ f1 = function(x) pmax(pmin((l + (u - l)/(1 + (2^x/ec50)^h)), 1, na.rm = T), 0, na.rm = T) ## slope sign convention of the drc package
+ return(tryCatch(integrate(f1, log2(md), log2(MD))$value/(log2(MD/md)),
+ error = function(e) {print(e); NA}))
+}
+
+# log IC50 from given dose-response parameters
+compute_log_ic50 <- function(l, u, ec50, h, md, MD) {
+ if((l >= 0.5) | (u <= 0.5)) {
+ return(NA)
+ } else {
+ f1 = function(x) (l + (u - l)/(1 + (2^x/ec50)^h) - 0.5) ## slope sign convention of the drc package
+ return(tryCatch(uniroot(f1, c(log2(md), log2(MD)))$root,
+ error = function(x) NA))
+ }
+}
+
+
+compute_MSE_MAD <- function(LFC, UL, LL, Slope, Inflection,
+ FC_column="FC", dose_column="dose") {
+ mse_compute <- LFC %>%
+ dplyr::filter(is.finite(.[[FC_column]]),is.finite(.[[dose_column]]) ) %>% ## in case there are some na values accidentally passed.
+ dplyr::mutate(FC.pred = UL + (LL -UL )/(1 + (.[[dose_column]]/Inflection)^Slope) ) %>% ## slope sign convention of the dr4pl package
+ dplyr::mutate(squared_deviation = (.[[FC_column]]-FC.pred)^2, abs_deviation = abs(.[[FC_column]]-FC.pred)) %>%
+ dplyr::summarise(mse = mean(squared_deviation,na.rm=T), mad= median(abs_deviation,na.rm=T))
+ return (mse_compute)
+}
+
+
+get_best_fit <- function(LFC_filtered, dose_var,
+ UL_low=0.8, UL_up=1.01, slope_decreasing=TRUE){
+ ## get best fit among different attempts at fitting, and see if this fit works sufficiently well to be reported.
+
+ LFC_filtered %<>% dplyr::filter(is.finite(FC),is.finite(.[[dose_var]]) ) ## remove any NA that may appear in the data
+ var_data <- LFC_filtered$FC %>% var()
+ riemann_AUC <- pmin(LFC_filtered$FC,1) %>% mean() ## mean fold-change after rounding FC to 1.
+ all_fits.df <- fit_4param_drc(LFC_filtered, dose_var, var_data,
+ UL_low, UL_up, slope_decreasing)
+
+ res.df <- data.frame(successful_fit=FALSE, auc_riemann = riemann_AUC) ## default return value if fit is unsuccessful
+
+
+
+ if (nrow(all_fits.df)>0){all_fits.df %<>% dplyr::filter(!is.na(frac_var_explained))} ## remove entries with NA in variance explained
+
+
+ if (nrow(all_fits.df)>0){
+ res.df <- all_fits.df %>%
+ slice_max(frac_var_explained, n = 1, with_ties = FALSE) %>% ## return best fit. if tied, return first of the ties
+ dplyr::mutate(successful_fit = TRUE,
+ auc_riemann = as.numeric(riemann_AUC) ) ## fit has to be at least as good as predicting just the mean of the data to be called successful
+ }
+
+ return (res.df)
+}
+
+fit_4param_drc <- function(LFC_filtered, dose_var, var_data,
+ UL_low=0.8, UL_up=1.01, slope_decreasing=TRUE) {
+ #fits a number of alternate models to the DRC and passes the results to the calling function (which chooses the best fit.)
+
+ # UL low is the lowerbound of UL we pass to the optimizer and UL_up is the upper bound of UL that we pass to the optimizer
+ # fomat of output will be:-
+ # results.df <- data.frame("fit_name"=character(),"Lower_Limit"=double(),
+ # "Upper_Limit"=double(),
+ # "Slope"=double(),
+ # "Inflection"=double(),
+ # "MSE"=double(), "MAD" =double(),
+ # "frac_var_explained"=double(),
+ # "Input_Parameters"=character())
+
+ results.df <- list(); ix = 1
+ slope_bound <- ifelse(slope_decreasing, 1e-5, Inf) # bound the slopes by default unless passed another option
+
+ dr4pl_initL <- tryCatch(dr4pl(as.formula(paste("FC ~ ", dose_var)), data = LFC_filtered,
+ init.parm = dr4pl::dr4pl_theta(theta_1 = 1, theta_4 = 0.005),
+ method.init = "logistic",
+ lowerl = c(UL_low, -Inf, -Inf, 0),
+ upperl = c(UL_up, Inf, slope_bound, 1.01)),
+ error= function(e){return(list(convergence=FALSE, error=TRUE))}
+ )
+ if (dr4pl_initL$convergence==FALSE){
+ if (!is.null(dr4pl_initL$dr4pl.robust)) {
+ dr4pl_initL <- dr4pl_initL$dr4pl.robust
+ }
+ }
+
+ if (dr4pl_initL$convergence){
+ mse_df <- compute_MSE_MAD(LFC_filtered, dr4pl_initL$parameters[[1]], dr4pl_initL$parameters[[4]],
+ dr4pl_initL$parameters[[3]], dr4pl_initL$parameters [[2]],
+ "FC", dose_var)
+
+ results.df[[ix]] <- tibble( fit_name = "dr4pl_initL",
+ Lower_Limit = as.numeric(dr4pl_initL$parameters [[4]]),
+ Upper_Limit = as.numeric(dr4pl_initL$parameters [[1]]),
+ Slope = as.numeric(dr4pl_initL$parameters [[3]]),
+ Inflection = as.numeric(dr4pl_initL$parameters [[2]]),
+ MSE = mse_df$mse, MAD = mse_df$mad, frac_var_explained = 1-mse_df$mse/var_data,
+ Input_Parameters = "constrained|init_logistic")
+ ix = ix + 1
+
+ }
+
+
+ dr4pl_initM_optNM <- tryCatch(dr4pl(as.formula(paste("FC ~ ", dose_var)), data = LFC_filtered,
+ init.parm = dr4pl::dr4pl_theta(theta_1 = 1, theta_4 = 0.01),
+ method.init = "Mead",
+ lowerl = c(UL_low, -Inf, -Inf, 0),
+ upperl = c(UL_up, Inf, slope_bound, 1.01),
+ method.optim="Nelder-Mead"),
+ error= function(e){return(list(convergence=FALSE, error=TRUE))}
+ )
+ if (dr4pl_initM_optNM$convergence==FALSE){
+ if (!is.null(dr4pl_initM_optNM$dr4pl.robust)) {
+ dr4pl_initM_optNM <- dr4pl_initM_optNM$dr4pl.robust
+ }
+ }
+
+ if (dr4pl_initM_optNM$convergence){
+ mse_df <- compute_MSE_MAD(LFC_filtered, dr4pl_initM_optNM$parameters[[1]], dr4pl_initM_optNM$parameters[[4]],
+ dr4pl_initM_optNM$parameters[[3]], dr4pl_initM_optNM$parameters [[2]],
+ "FC", dose_var)
+
+ results.df[[ix]] <- tibble( fit_name = "dr4pl_initM_optNM",
+ Lower_Limit = as.numeric(dr4pl_initM_optNM$parameters [[4]]),
+ Upper_Limit = as.numeric(dr4pl_initM_optNM$parameters [[1]]),
+ Slope = as.numeric(dr4pl_initM_optNM$parameters [[3]]),
+ Inflection = as.numeric(dr4pl_initM_optNM$parameters [[2]]),
+ MSE = mse_df$mse, MAD = mse_df$mad, frac_var_explained = 1-mse_df$mse/var_data,
+ Input_Parameters = "constrained|init_Mead|optim_Nelder-Mead")
+ ix = ix + 1
+ }
+
+
+ dr4pl_initL_optB <- tryCatch(dr4pl(as.formula(paste("FC ~ ", dose_var)), data = LFC_filtered,
+ init.parm = dr4pl::dr4pl_theta(theta_1 = 1, theta_4 = 0.01),
+ method.init = "logistic",
+ lowerl = c(UL_low, -Inf, -Inf, 0),
+ upperl = c(UL_up, Inf, slope_bound, 1.01),
+ method.optim="BFGS"),
+ error= function(e){return(list(convergence=FALSE, error=TRUE))}
+ )
+ if (dr4pl_initL_optB$convergence==FALSE){
+ if (!is.null(dr4pl_initL_optB$dr4pl.robust)) {
+ dr4pl_initL_optB <- dr4pl_initL_optB$dr4pl.robust
+ }
+ }
+ if (dr4pl_initL_optB$convergence){
+ mse_df <- compute_MSE_MAD(LFC_filtered, dr4pl_initL_optB$parameters[[1]], dr4pl_initL_optB$parameters[[4]],
+ dr4pl_initL_optB$parameters[[3]], dr4pl_initL_optB$parameters [[2]],
+ "FC", dose_var)
+
+
+ results.df[[ix]] <- tibble( fit_name = "dr4pl_initL_optB",
+ Lower_Limit = as.numeric(dr4pl_initL_optB$parameters [[4]]),
+ Upper_Limit = as.numeric(dr4pl_initL_optB$parameters [[1]]),
+ Slope = as.numeric(dr4pl_initL_optB$parameters [[3]]),
+ Inflection = as.numeric(dr4pl_initL_optB$parameters [[2]]),
+ MSE = mse_df$mse, MAD = mse_df$mad, frac_var_explained = 1-mse_df$mse/var_data,
+ Input_Parameters = "constrained|init_logistic|optim_BFGS")
+ ix = ix + 1
+
+ }
+
+ dr4pl_lossHuber<- tryCatch(dr4pl(as.formula(paste("FC ~ ", dose_var)), data = LFC_filtered,
+ init.parm = dr4pl::dr4pl_theta(theta_1 = 1, theta_4 = 0.01),
+ method.robust="Huber",
+ lowerl = c(UL_low, -Inf, -Inf, 0),
+ upperl = c(UL_up, Inf, slope_bound, 1.01)),
+ error= function(e){return(list(convergence=FALSE, error=TRUE))}
+ )
+ if (dr4pl_lossHuber$convergence==FALSE){
+ if (!is.null(dr4pl_lossHuber$dr4pl.robust)) {
+ dr4pl_lossHuber <- dr4pl_lossHuber$dr4pl.robust
+ }
+ }
+ if (dr4pl_lossHuber$convergence){
+ mse_df <- compute_MSE_MAD(LFC_filtered, dr4pl_lossHuber$parameters[[1]], dr4pl_lossHuber$parameters[[4]],
+ dr4pl_lossHuber$parameters[[3]], dr4pl_lossHuber$parameters [[2]],
+ "FC", dose_var)
+
+
+ results.df[[ix]] <- tibble( fit_name = "dr4pl_lossHuber",
+ Lower_Limit = as.numeric(dr4pl_lossHuber$parameters [[4]]),
+ Upper_Limit = as.numeric(dr4pl_lossHuber$parameters [[1]]),
+ Slope = as.numeric(dr4pl_lossHuber$parameters [[3]]),
+ Inflection = as.numeric(dr4pl_lossHuber$parameters [[2]]),
+ MSE = mse_df$mse, MAD = mse_df$mad, frac_var_explained = 1-mse_df$mse/var_data,
+ Input_Parameters = "constrained|loss_Huber")
+ ix = ix + 1
+
+ }
+
+ ### add in original default drc into pipeline just to compare. ######
+ dr4pl_unconstrained <- tryCatch(dr4pl(dose = LFC_filtered[[dose_var]], response = LFC_filtered$FC,
+ init.parm = dr4pl::dr4pl_theta(theta_1 = 1, theta_4 = 0.3),
+ method.init = "logistic",
+ lowerl = c(0.99, -Inf, -Inf, 0),
+ upperl = c(1.01, Inf, Inf, 1.01)),
+ error = function(e) {print(e); return(NA)})
+
+ # if it fits and doesn't converge grab robust fit
+ if (!all(is.na(dr4pl_unconstrained))) {
+ if (!dr4pl_unconstrained$convergence) {
+ dr4pl_unconstrained <- dr4pl_unconstrained$dr4pl.robust
+ }
+ }
+ # get parameters
+ param <- tryCatch(dr4pl_unconstrained$parameters, error = function(e) return(NA))
+ if (!all(is.na(param))){
+ if(as.numeric(dr4pl_unconstrained$parameters [[3]])
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+#RUN echo "deb http://security.debian.org/debian-security bullseye-security main contrib non-free" > /etc/apt/sources.list
+#RUN sed -i 's/deb.debian.org/archive.debian.org/' /etc/apt/sources.list
+#RUN apt-get update
+RUN apt-get install -y jq
+COPY ./eps-prep-bash.sh /clue/bin/eps-prep-bash
+COPY ./eps-prep.py /clue/bin/eps-prep.py
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+
+WORKDIR /
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["eps-prep-bash"]
diff --git a/eps-prep/build_docker.sh b/eps-prep/build_docker.sh
new file mode 100755
index 0000000..8ee6c40
--- /dev/null
+++ b/eps-prep/build_docker.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/eps-prep:develop --rm=true .
+
+docker push prismcmap/eps-prep:develop
\ No newline at end of file
diff --git a/eps-prep/docker_push_tags.sh b/eps-prep/docker_push_tags.sh
new file mode 100755
index 0000000..cd52cfe
--- /dev/null
+++ b/eps-prep/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/merge-csv-files:v0.0.2
+docker push prismcmap/merge-csv-files:production
+
+# trigger
\ No newline at end of file
diff --git a/eps-prep/eps-prep-bash.sh b/eps-prep/eps-prep-bash.sh
new file mode 100755
index 0000000..a258f78
--- /dev/null
+++ b/eps-prep/eps-prep-bash.sh
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+#setup environment
+source activate prism
+
+#return to /
+cd /
+
+print_help () {
+ python /clue/bin/eps-prep.py --help
+ echo " -spl, --search_pattern_list - comma separated list of search patterns in DRC, e.g. 'DRC_TABLE*,discrete_associations*' (bash)"
+}
+
+#optional
+if test $# -lt 1; then
+ print_help
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit 0
+ ;;
+ -f|--compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -d|--data_dir)
+ shift
+ DATA_DIR=$1
+ ;;
+ -o|--out)
+ shift
+ OUT_DIR=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+batch_index=0
+if [[ ! -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+fi
+
+args=()
+
+if [[ ! -z $projects ]]
+then
+ PROJECT=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ DATA_DIR="${DATA_DIR}"/"${PROJECT,,}"/
+fi
+
+
+args+=(
+ -d "$DATA_DIR"
+)
+
+echo python /clue/bin/eps-prep.py "${args[@]}"
+python /clue/bin/eps-prep.py "${args[@]}"
+
+exit_code=$?
+conda deactivate
+exit $exit_code
diff --git a/eps-prep/eps-prep.py b/eps-prep/eps-prep.py
new file mode 100644
index 0000000..95e0e3f
--- /dev/null
+++ b/eps-prep/eps-prep.py
@@ -0,0 +1,84 @@
+"""
+Module takes in a project directory for an Extended Day build and adapts it for EPS downloads
+
+1. Removes all ic50 references
+"""
+
+import argparse
+import glob
+import logging
+import os
+import sys
+import pandas as pd
+
+logger = logging.getLogger('eps-prep')
+biomarker_search_patterns = [
+ '*continuous_association*',
+ '*discrete_associations*',
+ '*RF_table*',
+ '*model_table*'
+]
+
+
+def build_parser():
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ parser.add_argument('--project_dir', '-d', help="Project Directory", type=str, required=True)
+ return parser
+
+
+"""
+Drops the ic50 column from the DRC_TABLE file, replaces file in place
+"""
+def remove_ic50_from_drc_table(project_dir):
+ search_path = os.path.join(project_dir, "*", "data", "*DRC_TABLE*")
+ file = glob.glob(search_path)
+ assert(len(file) == 1), f"Expected to find one file matching pattern: {search_path}"
+ df = pd.read_csv(file[0])
+ if 'log2.ic50' in df.columns:
+ df = df.drop(columns=['log2.ic50'])
+ df.to_csv(file[0], index=False)
+ return
+ else:
+ print("IC50 columns not present")
+
+
+"""
+Deletes the IC50_MATRIX file
+"""
+def delete_ic50_matrix_file(project_dir):
+ search_path = os.path.join(project_dir, "*", "data", "*IC50_MATRIX*")
+ file = glob.glob(search_path)
+ if len(file) == 0:
+ print("No IC50_MATRIX file found, skipping")
+ return
+ else:
+ assert(len(file) == 1), f"Expected to find one file matching pattern: {search_path}"
+ os.remove(file[0])
+ return
+
+"""
+Removes all ic50 dose rows from biomarker files
+"""
+def remove_ic50_doses_from_biomarker(project_dir):
+ for sp in biomarker_search_patterns:
+ # matching project_dir/{PROJECT
+ file = glob.glob(os.path.join(project_dir, "*", "data", sp))
+ print(file)
+ assert(len(file) == 1), "Expected to find one file matching pattern: {}".format(sp)
+ df = pd.read_csv(file[0])
+ # remove rows where pert_dose = 'log2.ic50' using .loc
+ df = df.loc[df['pert_dose'] != 'log2.ic50']
+ df.to_csv(file[0], index=False)
+ return
+
+
+def main(args):
+ remove_ic50_from_drc_table(args.project_dir)
+ delete_ic50_matrix_file(args.project_dir)
+ remove_ic50_doses_from_biomarker(args.project_dir)
+ return
+
+if __name__ == "__main__":
+ args = build_parser().parse_args(sys.argv[1:])
+ main(args)
+
diff --git a/extract-biomarkers/.dockerignore b/extract-biomarkers/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/extract-biomarkers/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/extract-biomarkers/DEPRECATED_Dockerfile b/extract-biomarkers/DEPRECATED_Dockerfile
new file mode 100644
index 0000000..8c19e61
--- /dev/null
+++ b/extract-biomarkers/DEPRECATED_Dockerfile
@@ -0,0 +1,16 @@
+FROM python:latest
+MAINTAINER Jacob Asiedu
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+RUN apt-get update -qq && \
+ apt-get install -y jq && \
+ pip install pandas
+COPY ./extract_biomarker_bash.sh /clue/bin/extract_biomarker_bash
+COPY ./extract_biomarker.py /clue/bin/extract_biomarker.py
+
+WORKDIR /
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["extract_biomarker_bash"]
+
diff --git a/extract-biomarkers/README.md b/extract-biomarkers/README.md
new file mode 100644
index 0000000..f28cf79
--- /dev/null
+++ b/extract-biomarkers/README.md
@@ -0,0 +1,2 @@
+This module is for internal use only. It is used to extract from continous_associations.csv, biomarker file per compound per project.
+Then it creates different files by extracting the top 10 biomakers from the first set of files
\ No newline at end of file
diff --git a/extract-biomarkers/build_docker.sh b/extract-biomarkers/build_docker.sh
new file mode 100755
index 0000000..3f12ae0
--- /dev/null
+++ b/extract-biomarkers/build_docker.sh
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/extract-biomarker:latest -t prismcmap/extract-biomarker:v0.0.1 --rm=true .
diff --git a/extract-biomarkers/docker_push_tags.sh b/extract-biomarkers/docker_push_tags.sh
new file mode 100755
index 0000000..b1476f3
--- /dev/null
+++ b/extract-biomarkers/docker_push_tags.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+docker push prismcmap/extract-biomarker:v0.0.1
+docker push prismcmap/extract-biomarker:latest
+
+# trigger
\ No newline at end of file
diff --git a/extract-biomarkers/docker_run.sh b/extract-biomarkers/docker_run.sh
new file mode 100755
index 0000000..9f8a9e8
--- /dev/null
+++ b/extract-biomarkers/docker_run.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+docker run --rm \
+--name bar2 \
+-v /Users/jasiedu/WebstormProjects/TEST_CPS017_P1000_NEWFMT3:/cmap/macchiato \
+-e projects="$(cat /Users/jasiedu/WebstormProjects/macchiato/foo1.json)" \
+-e AWS_BATCH_JOB_ARRAY_INDEX=0 \
+-it prismcmap/extract-biomarker \
+-d /cmap/macchiato/projects \
+-o /cmap/macchiato/projects \
+-x 10
\ No newline at end of file
diff --git a/extract-biomarkers/extract_biomarker.py b/extract-biomarkers/extract_biomarker.py
new file mode 100644
index 0000000..9ea0410
--- /dev/null
+++ b/extract-biomarkers/extract_biomarker.py
@@ -0,0 +1,36 @@
+import argparse
+import os
+import sys
+
+import pandas as pd
+
+# read DataFrame
+def build_parser():
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+ # The following arguments are required. These are files that are necessary for top_10_biomarker
+ parser.add_argument("--data_dir", "-d", help="Required: Path to continuous association file",type=str, required=True)
+ parser.add_argument("--out_dir", "-o", help="Required: out folder",type=str, required=True)
+ parser.add_argument("--extract_top_x", "-x", help="Extract the top x results",type=str, default="10")
+ parser.add_argument("--verbose", '-v', help="Whether to print a bunch of output", action="store_true", default=False)
+ return parser
+
+def build(args):
+ top_x = args.extract_top_x
+ if os.path.exists(args.data_dir):
+ data = pd.read_csv(args.data_dir)
+ top_x_biomarker_file_path =os.path.join(args.out_dir,'top_' + top_x + '_biomarkers.csv')
+ data.reindex(data.coef.abs().sort_values(ascending=False).index)
+ df = data.head(int(top_x))
+ df.to_csv(top_x_biomarker_file_path, index=False)
+ return
+
+def main(args):
+ try:
+ build(args)
+ except Exception as e:
+ print(e)
+
+if __name__ == "__main__":
+ args = build_parser().parse_args(sys.argv[1:])
+ print(args)
+ main(args)
diff --git a/extract-biomarkers/extract_biomarker_bash.sh b/extract-biomarkers/extract_biomarker_bash.sh
new file mode 100755
index 0000000..0f96fc1
--- /dev/null
+++ b/extract-biomarkers/extract_biomarker_bash.sh
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+
+#optional
+if test $# -lt 1; then
+ printf "Usage ./extract_biomarker_bash.sh [options]\nOptions include:\n"
+ printf -- "\t-d, --data_dir \t Path to the continuous association folder (required) \n"
+ printf -- "\t-o, --out_dir \t Output folder for top 10 files (required) \n"\
+ printf -- "\t-x, --extract_top_x \t The number of top biomarkers to extract from file (defaults to 10) \n"
+ printf -- "\t-h, --help \t\t Print this help text\n"
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ printf "Usage ./extract_biomarker_bash.sh [options]\nOptions include:\n"
+ printf -- "\t-d, --data_dir \t Path to the continuous association folder (required) \n"
+ printf -- "\t-o, --out_dir \t Output folder for top X files (required) \n"
+ printf -- "\t-x, --extract_top_x \t The number of top biomarkers to extract from file (defaults to 10) \n"
+ printf -- "\t-h, --help \t\t Print this help text\n"
+ exit 0
+ ;;
+ -d| --data_dir)
+ shift
+ DATA_DIR=$1
+ ;;
+ -f| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -o| --out_dir)
+ shift
+ OUT_DIR=$1
+ ;;
+ -x| --extract_top_x)
+ shift
+ TOP_X=$1
+ ;;
+ -v| --verbose)
+ shift
+ VERBOSE=true
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+if [ ! -d $OUT_DIR ]
+then
+ mkdir -p $OUT_DIR
+fi
+
+##Extract Top X biomarkers, defaults to 10
+if [[ -z $TOP_X ]]
+then
+ TOP_X=10
+fi
+
+if [[ -z $DATA_DIR || -z $OUT_DIR ]]
+then
+ printf "Required arguments missing\n"
+ exit -1
+fi
+
+
+
+#return to /
+cd /
+
+batch_index=0
+
+
+if [[ ! -z $projects ]]
+then
+ #If this is an array job set batch_index
+ if [[ ! -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+ fi
+ pert_id=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_id')
+ project=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ plate=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].pert_plate')
+ cleaned_pert_id=$(echo "${pert_id//|/$'_'}")
+ sanitized_pert_id="${cleaned_pert_id^^}"
+ DATA_DIR="${DATA_DIR}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"/continuous_associations.csv
+ OUT_DIR="${OUT_DIR}"/"${project,,}"/"${project^^}"/"${plate}"/"${sanitized_pert_id}"
+fi
+
+args=(
+ -d "$DATA_DIR"
+ -o "$OUT_DIR"
+ -x "$TOP_X"
+)
+
+if [[ ! -z $VERBOSE ]]
+then
+ args+=(-v)
+fi
+echo -d "${DATA_DIR}" -o "${OUT_DIR}" -x "${TOP_X}"
+
+if [[ "$pert_id" == "DMSO" ]]
+then
+ echo "Skipping DMSO"
+else
+ python /clue/bin/extract_biomarker.py "${args[@]}"
+fi
+
+
+exit_code=$?
+exit $exit_code
\ No newline at end of file
diff --git a/filter-skipped-wells/.dockerignore b/filter-skipped-wells/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/filter-skipped-wells/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/filter-skipped-wells/Dockerfile b/filter-skipped-wells/Dockerfile
new file mode 100644
index 0000000..86248f2
--- /dev/null
+++ b/filter-skipped-wells/Dockerfile
@@ -0,0 +1,18 @@
+FROM prismcmap/base-prism:latest
+MAINTAINER Anup Jonchhe
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+RUN apt-get install -y jq
+COPY ./aws_batch.sh /clue/bin/filter_skipped_wells
+COPY ./filter_skipped_wells.py /clue/bin/filter_skipped_wells.py
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+
+#RUN cd /cmap && git clone https://github.com/cmap/merino.git
+
+WORKDIR /
+
+ENV PATH /clue/bin:$PATH
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENTRYPOINT ["filter_skipped_wells"]
diff --git a/filter-skipped-wells/README.md b/filter-skipped-wells/README.md
new file mode 100644
index 0000000..391f13c
--- /dev/null
+++ b/filter-skipped-wells/README.md
@@ -0,0 +1,67 @@
+# Deal Module - Testing Jenkins
+
+## Deal Module Overview
+
+Distributes files to make build for each project
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/deal) run:
+
+```
+docker pull prismcmap/deal:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `deal_bash.sh` which is a wrapper around `deal.py` for more information using python run
+
+```
+$ python deal.py --help
+usage: deal.py [-h] [--build_path BUILD_PATH] [--only_key ONLY_KEY]
+ [--project PROJECT] [--sig_id_cols SIG_ID_COLS] [--out OUT]
+ [--verbose]
+
+Distributes files to make build for each project
+
+optional arguments:
+ -h, --help show this help message and exit
+ --build_path BUILD_PATH, -b BUILD_PATH
+ Build path (default: None)
+ --only_key ONLY_KEY, -k ONLY_KEY
+ key to extract. Useful if parallelizing, only listed
+ keys will be concatenated (default: None)
+ --project PROJECT, -p PROJECT
+ Project to extract (default: None)
+ --sig_id_cols SIG_ID_COLS, -s SIG_ID_COLS
+ Comma separated list of col names to create sig_ids if
+ not present (default:
+ pert_plate,culture,pert_id,pert_idose,pert_time)
+ --out OUT, -o OUT Output for collated build (default: None)
+ --verbose, -v Whether to print a bunch of output (default: False)
+
+```
+
+### Example usage with Docker
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+
+```
+docker run \
+ -it \
+ -v ~/{PROJECT_DIR}:/data/ \
+ prismcmap/deal:latest \
+ -b /data/build \
+ -o /data/projects
+```
+
+
+### Example usage with Python
+
+Running python requires an environment set up to use merino. See merino repo here: https://github.com/cmap/merino
+
+```
+python deal.py -b {PROJECT_DIR}/build -o {PROJECT_DIR}/projects
+```
diff --git a/filter-skipped-wells/aws_batch.sh b/filter-skipped-wells/aws_batch.sh
new file mode 100644
index 0000000..d64ae4b
--- /dev/null
+++ b/filter-skipped-wells/aws_batch.sh
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+source activate prism
+
+print_help () {
+ python /clue/bin/filter_skipped_wells.py --help
+}
+
+#optional
+if test $# -lt 1; then
+ print_help
+ exit 1
+fi
+
+while test $# -gt 0; do
+ case "$1" in
+ -h|--help)
+ print_help
+ exit 0
+ ;;
+ -d| --data_dir)
+ shift
+ DATA_DIR=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ ;;
+ esac
+ shift
+done
+
+if [[ -z $DATA_DIR ]]
+then
+ printf "Required arguments missing\n"
+ exit -1
+fi
+
+args=(
+ -d "$DATA_DIR"
+)
+python /clue/bin/filter_skipped_wells.py "${args[@]}"
+
+exit_code=$?
+
+echo "$exit_code"
+exit $exit_code
\ No newline at end of file
diff --git a/filter-skipped-wells/build_docker.sh b/filter-skipped-wells/build_docker.sh
new file mode 100755
index 0000000..9c64a98
--- /dev/null
+++ b/filter-skipped-wells/build_docker.sh
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t prismcmap/filter-skipped-wells:develop --rm=true .
+
+
+
diff --git a/filter-skipped-wells/docker_push_tags.sh b/filter-skipped-wells/docker_push_tags.sh
new file mode 100755
index 0000000..7e9977d
--- /dev/null
+++ b/filter-skipped-wells/docker_push_tags.sh
@@ -0,0 +1,2 @@
+#!/usr/bin/env bash
+docker push prismcmap/filter-skipped-wells:develop
\ No newline at end of file
diff --git a/filter-skipped-wells/filter_skipped_wells.py b/filter-skipped-wells/filter_skipped_wells.py
new file mode 100644
index 0000000..4fda348
--- /dev/null
+++ b/filter-skipped-wells/filter_skipped_wells.py
@@ -0,0 +1,168 @@
+import glob
+import numpy as np
+import pandas as pd
+import argparse
+import requests
+import os
+# import boto3
+import gzip
+import io
+
+def make_request_url_filter(endpoint_url, where=None, fields=None):
+ clauses = []
+ if where:
+ where_clause = '"where":{'
+ wheres = []
+ for k, v in where.items():
+ print(type(v))
+ if isinstance(v, (list, np.ndarray)):
+ wheres.append('"{k}":{{"inq": ["{v}"]}}'.format(k=k, v='","'.join(v)))
+ else:
+ wheres.append('"{k}":"{v}"'.format(k=k, v=v))
+ where_clause += ','.join(wheres) + '}'
+ print(where_clause)
+ clauses.append(where_clause)
+
+ if fields:
+ fields_clause = '"fields":{'
+ fields_list = []
+ if type(fields) == dict:
+ for k, v in fields.items():
+ fields_list.append('"{k}":"{v}"'.format(k=k, v=v))
+ elif type(fields) == list:
+ for field in fields:
+ fields_list.append('"{k}":"{v}"'.format(k=field, v="true"))
+ fields_clause += ','.join(fields_list) + '}'
+ clauses.append(fields_clause)
+
+ if len(clauses) > 0:
+ # print(endpoint_url.rstrip("/") + '?filter={' + ','.join(clauses) + '}')
+ return endpoint_url.rstrip("/") + '?filter={' + requests.utils.quote(','.join(clauses)) + '}'
+ else:
+ return endpoint_url
+
+
+def get_data_from_db(endpoint_url, user_key, where=None, fields=None):
+ request_url = make_request_url_filter(endpoint_url, where=where, fields=fields)
+ print(request_url)
+ response = requests.get(request_url, headers={'user_key': user_key, 'prism_key': 'prism_mts'})
+ if response.ok:
+ return response.json()
+ else:
+ response.raise_for_status()
+
+
+def load_df_from_s3(partial_filename, prefix, bucket_name='macchiato.clue.io'):
+ s3 = boto3.client('s3')
+
+ # List objects in the bucket with the given prefix
+ objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
+
+ # Find the file that contains the partial filename
+ matching_file = None
+ for obj in objects.get('Contents', []):
+ if partial_filename in obj['Key']:
+ matching_file = obj['Key']
+ break
+
+ if not matching_file:
+ raise ValueError(f"No file found with partial name: {partial_filename}")
+
+ # Get the object from S3
+ response = s3.get_object(Bucket=bucket_name, Key=matching_file)
+ gzipped_bytes = response['Body'].read()
+
+ # Decompress the gzipped bytes
+ with gzip.GzipFile(fileobj=io.BytesIO(gzipped_bytes)) as gz:
+ df = pd.read_csv(gz)
+
+ return df
+
+
+def process_data(data, plates):
+ sw_data = pd.DataFrame(get_data_from_db(endpoint_url=SW_URL,
+ user_key=API_KEY,
+ where={"pert_plate": plates}))
+
+ # Check if sw_data is empty, and if it is, print a warning and return
+ if sw_data.empty:
+ print(f"Warning: No matching records returned for plates {plates}. No merge performed. This is fine if there are no expected skipped wells.")
+ return data, pd.DataFrame() # Return the original data and an empty DataFrame for filtered_out_values
+
+ # Rename columns if they exist
+ sw_data.rename(columns={'assay_well_position': 'pert_well'}, inplace=True)
+
+ # Columns to match on
+ cols_to_match = ['screen', 'pert_plate', 'pert_well', 'pool_id', 'replicate']
+
+ # Check if all cols_to_match are in sw_data columns
+ if not all(col in sw_data.columns for col in cols_to_match):
+ print(f"Warning: Not all columns to match are present in the sw_data. No merge performed.")
+ return data, pd.DataFrame()
+
+ # Merge the two dataframes on the specified columns with the indicator argument
+ merged = data.merge(sw_data[cols_to_match],
+ on=cols_to_match,
+ how='left',
+ indicator=True)
+
+ # Filter out the rows that are present in both dataframes
+ level3_data_filtered = merged[merged['_merge'] == 'left_only'].drop(columns=['_merge'])
+
+ # Filter the rows that are present in both dataframes to get the values that are filtered out
+ filtered_out_values = merged[merged['_merge'] == 'both'].drop(columns=['_merge'])
+
+ return level3_data_filtered, filtered_out_values
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description="Take a data_dir and screen_id and filter out skipped Echo wells."
+ "Return a pruned level3 df for use in the MTS pipeline.")
+ # parser.add_argument("-b", "--build", help="Name of the build on S3.")
+ parser.add_argument("-d", "--data_dir", help="Name of the build on S3.", required=True)
+ # parser.add_argument("-s", "--screen", help="Name of the screen.", required=True)
+ # parser.add_argument("-s", "--search_pattern", help="search path of file in --data_dir to remove skipped wells for",
+ # required=True, default='*LEVEL3_LMFI.csv')
+
+ args = parser.parse_args()
+ print(args)
+
+ if os.environ.get('API_KEY'):
+ API_KEY = os.environ['API_KEY']
+ else:
+ print("API_KEY required")
+ exit(1)
+
+
+ if os.environ.get('API_URL'):
+ API_URL = os.environ['API_URL']
+ else:
+ API_URL = 'https://api.clue.io/api/'
+
+ if not API_URL.rstrip("/").endswith("api/"):
+ API_URL = API_URL.rstrip("/") + "/api/"
+
+ SW_URL = API_URL.rstrip("/") + '/v_assay_plate_skipped_well/'
+ # BUCKET_NAME = 'macchiato.clue.io'
+
+ search_pattern = "*LEVEL3_LMFI.csv"
+
+ fps = glob.glob(os.path.join(args.data_dir, "*LEVEL3_LMFI.csv"))
+
+ if len(fps) != 1:
+ raise ValueError(f"Expected 1 file to match {search_pattern} but found {len(fps)}")
+
+ data = pd.read_csv(fps[0]);
+ plates = data['pert_plate'].unique()
+
+ level3_data_filtered, filtered_out_values = process_data(data=data, plates=plates)
+
+ # Write the filtered data to a csv
+ level3_data_filtered.to_csv(fps[0], index=False)
+
+ # Check if filtered_out_values is not empty before writing
+ if not filtered_out_values.empty:
+ filtered_out_values.to_csv("removed_wells.csv", index=False)
+ else:
+ print("No wells were removed. 'removed_wells.csv' was not created.")
+
diff --git a/get_release_diffs.sh b/get_release_diffs.sh
new file mode 100755
index 0000000..228fdc0
--- /dev/null
+++ b/get_release_diffs.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+MAIN_BRANCH="master"
+DEVELOP_BRANCH="develop"
+
+# Define the function to get_changes() here
+get_changes() {
+ local changes
+ changes=$(git log --oneline --no-merges ${MAIN_BRANCH}..${DEVELOP_BRANCH} -- "$1")
+ echo -n "$changes" # Preserve newline characters
+}
+
+rm changes.txt
+touch changes.txt
+
+# Iterate through each folder in the current directory
+for folder in */; do
+ folder="${folder%/}" # Remove trailing slash
+ if [ -d "$folder" ]; then
+ CHANGES=$(get_changes "$folder")
+ if [[ ! -z "$CHANGES" ]]; then
+ echo "$folder" >> changes.txt
+ echo -e "$CHANGES" >> changes.txt
+ echo "" >> changes.txt
+ echo "" >> changes.txt
+ fi
+ fi
+done
diff --git a/info/ColumnHeaders.md b/info/ColumnHeaders.md
new file mode 100644
index 0000000..69c7564
--- /dev/null
+++ b/info/ColumnHeaders.md
@@ -0,0 +1,174 @@
+# Column Headers
+
+Bolded columns are especially important while unbolded columns are primarily for internal use or unused. Italicized columns will only appear for combination perturbations.
+
+### LEVEL2 through LEVEL5
+
+| Column | Description |
+|:-------|:------------|
+| prism_replicate | Sample plate replicate |
+| rid | Cell line ID |
+| profile_id | Concatenation of replicate and well |
+| **ccle_name** | Cell line name |
+| **pool_id** | Cell line pool |
+| **culture** | Cell line culture (e.g. PR500) |
+| mfc_plate_id | Plate ID (deprecated) |
+| mfc_plate_name | Plate name |
+| pert_dose | Perturbation dose (numeric) |
+| pert_dose_unit | Perturbation dose units |
+| **pert_id** | Perturbation Broad ID |
+| pert_idose | Perturbation dose with units |
+| **pert_iname** | Perturbation name (e.g. AZ-628) |
+| pert_itime | Assay length with units |
+| pert_mfc_desc | Perturbation description|
+| pert_mfc_id | Broad perturbation ID with batch suffix |
+| pert_mfc_plate | Plate barcode |
+| **pert_plate** | Sample compound plate |
+| **pert_time** | Assay length |
+| pert_time_unit | Assay length units |
+| pert_type | Perturbation type (e.g. negative control) |
+| **pert_vehicle** | Perturbation vehicle (e.g. DMSO) |
+| **pert_well** | Sample well |
+| x_group_by | Grouping variable for samples across replicates |
+| x_partner | Collaborator name |
+| x_pert_plate | pert_plate + x_replicate |
+| **x_project_id** | Project name (e.g. Validation Compounds) |
+| x_replicate | Plate replicate |
+| x_source_well | Sample source plate well |
+| is_well_failure | QC indicator (failures removed after LEVEL2)
+| **logMFI_norm** | logMFI normalized to control barcodes |
+| **logMFI** | log2 median fluorescence intensity |
+| feature_id | Unique feature ID |
+| **LFC** | log2 fold-change relative to control vehicle (this is median collapsed across replicates in LEVEL5) |
+| **LFC_cb** | ComBat corrected LFC (this is median collapsed across replicates in LEVEL5) |
+
+### QC\_TABLE
+
+| Column | Description |
+|:-------|:------------|
+| **prism_replicate** | Sample plate replicate |
+| **ccle_name** | Cell line name |
+| **pert_time** | Assay length |
+| rid | Cell line ID |
+| **pool_id** | Cell line pool |
+| **culture** | Cell line culture |
+| **pert_plate** | Sample compound plate |
+| ctl_vehicle_md | Control vehicle median logMFI |
+| trt_poscon_md | Positive control median logMFI |
+| ctl_vehicle_mad | Control vehicle MAD logMFI |
+| trt_poscon_mad | Positive control MAD logMFI |
+| ssmd | Strictly-standardized mean difference between positive and negative controls |
+| nnmd | Null-normalized mean difference between positive and negative controls |
+| **error_rate** | Error rate between positive and negative controls |
+| **dr** | Dynamic range (difference in control medians) |
+| **pass** | Boolean indicating whether cell line passed QC |
+
+### DRC\_TABLE
+
+| Column | Description |
+|:-------|:------------|
+| **min_dose** | Minimum dose of perturbation |
+| **max_dose** | Maximum dose of perturbation |
+| **upper_limit** | Upper limit of the curve |
+| **ec50** | Inflection point or relative IC50 of the curve |
+| **slope** | Slope of the dose-response curve |
+| **lower_limit** | Lower limit of the curve |
+| convergence | Did the fit converge? |
+| **auc** | Area under the curve |
+| **log2.ic50** | log2 of the absolute IC50 of the curve|
+| mse | Mean-squared error of the curve fit |
+| R2 | r-squared of the curve fit |
+| **varied_iname** | Perturbation name for the test perturbation |
+| **varied_id** | Broad ID for the test perturbation |
+| **ccle_name** | Cell line name |
+| **culture** | Cell line culture |
+| pert_time | Assay length |
+| pert_plate | Compound plate |
+| *added_compounds* | Perturbation name(s) for anchor compound(s) |
+| *added_doses* | Perturbation dose(s) for anchor compound(s) |
+| *added_ids* | Perturbation ID(s) for anchor compound(s) |
+
+### discrete\_associations
+
+| Column | Description |
+|:-------|:------------|
+| **effect_size** | Difference in means between groups |
+| **feature** | Feature defining groups |
+| **feature_type** | Type of feature |
+| p.value | p-value |
+| **pert_dose** | Perturbation dose |
+| **pert_id** | Perturbation ID |
+| **pert_iname** | Perturbation name |
+| pert_plate | Compound plate |
+| pert_time | Assay length |
+| **q.value** | q-value (corrected p-value) |
+| t_stat | t statistic |
+| *added_compounds* | Perturbation name(s) for anchor compound(s) |
+| *added_doses* | Perturbation dose(s) for anchor compound(s) |
+| *added_ids* | Perturbation ID(s) for anchor compound(s) |
+
+### continuous\_associations
+
+| Column | Description |
+|:-------|:------------|
+| NegativeProb | Posterior probability that beta is negative |
+| PositiveProb | Posterior probability that beta is positive |
+| PosteriorMean | Adaptive shrinkage moderated effect size estimate |
+| PosteriorSD | Standard deviation of the PosteriorMean
+| betahat | Effect size estimate |
+| **coef** | Correlation coefficient |
+| feature | Correlated feature |
+| feature_type | Type of correlated feature
+| lfdr | Local FDR value |
+| lfsr | Local FSR value |
+| p.val | p-value for correlation coefficient |
+| **pert_dose** | Perturbation dose |
+| **pert_id** | Perturbation ID |
+| **pert_iname** | Perturbation name |
+| pert_plate | Compound plate |
+| pert_time | Assay length |
+| **q.val** | q-value for correlation coefficient |
+| qvalue | q-value for beta |
+| rank | Feature rank |
+| sebetahat | Beta standard error |
+| svalue | s-value |
+| *added_compounds* | Perturbation name(s) for anchor compound(s) |
+| *added_doses* | Perturbation dose(s) for anchor compound(s) |
+| *added_ids* | Perturbation ID(s) for anchor compound(s) |
+
+### RF\_table
+
+| Column | Description |
+|:-------|:------------|
+| **RF.imp.mean** | Mean feature importance |
+| RF.imp.sd | Feature importance standard deviation |
+| RF.imp.stability | Fraction of trees using the feature |
+| **feature** | Feature name |
+| **model** | Dataset used for features |
+| **pert_dose** | Perturbation dose |
+| **pert_id** | Perturbation ID |
+| **pert_iname** | Perturbation name |
+| pert_plate | Compound plate |
+| pert_time | Assay length |
+| rank | Feature rank |
+| *added_compounds* | Perturbation name(s) for anchor compound(s) |
+| *added_doses* | Perturbation dose(s) for anchor compound(s) |
+| *added_ids* | Perturbation ID(s) for anchor compound(s) |
+
+### model\_table
+
+| Column | Description |
+|:-------|:------------|
+| MSE | Mean squared error of the model |
+| MSE.se | Standard error of the MSE |
+| **PearsonScore** | Pearson score of the model |
+| **R2** | r-squared of the model |
+| **model** | Dataset used for features |
+| **pert_dose** | Perturbation dose |
+| **pert_id** | Perturbation ID |
+| **pert_iname** | Perturbation name |
+| pert_plate | Compound plate |
+| pert_time | Assay length |
+| *added_compounds* | Perturbation name(s) for anchor compound(s) |
+| *added_doses* | Perturbation dose(s) for anchor compound(s) |
+| *added_ids* | Perturbation ID(s) for anchor compound(s) |
diff --git a/info/Dockerfile b/info/Dockerfile
new file mode 100644
index 0000000..28a91bd
--- /dev/null
+++ b/info/Dockerfile
@@ -0,0 +1,3 @@
+FROM busybox
+
+CMD [ "ps", "faux" ]
diff --git a/info/FAQs.md b/info/FAQs.md
new file mode 100644
index 0000000..d908231
--- /dev/null
+++ b/info/FAQs.md
@@ -0,0 +1,27 @@
+# FAQs
+Frequently asked questions about MTS data processing and reports. This document is continuously updated.
+
+### Which datasets were used for biomarker analysis?
+
+For external collaborators, the latest public datasets available on [depmap.org](https://depmap.org/portal/download) are used. For internal collaborators, the latest internal datasets are used (available on internal versions of the portal).
+
+### Why use error rate instead of SSMD as QC?
+
+We find that error rate is better at determining which cell lines are noisiest. If you wish to use SSMD to QC your data (for example to reflect previous PRISM results) the information is tabulated in `SSMD_TABLE.csv`. When using SSMD we suggest using a cutoff of 2. You can also find other metrics like NNMD and normalized logMFI values for positive and negative controls.
+
+### What are log-fold change values and how should I use them?
+
+Log-fold change (LFC) is the $\log_2$ of the ratio of cell line abundance in treatment versus DMSO (negative control), this ratio is also called viability (so viability = $2^{LFC}$). LFC values can vary based on a number of factors (cell line growth rate, compound potency, noise) but **in general we suggest using a viability of 0.3 as a cutoff for "killing" of a cell line** this corresponds to LFC <= $\log_2(0.3)$.
+
+### Why are mutation and lineage separate from other biomarkers?
+
+Lineage and mutation are discrete variables that describe a cell line and therefore require different statistical tests than continuous variables to quantify their effect (t-test as opposed to correlations). For more information on how biomarker analyses are done see our [analysis info document](./analysis_info.pdf).
+
+### How can I filter cell lines down to have more "trustworthy" lines?
+
+We only use error rate to determine whether cell lines pass QC (see the [pipeline description](./MTS_pipeline.md) for details). We do this to be as light-handed with the data as possible. To filter further we recommend a few different strategies:
+
+1. Filter by **SSMD**. Typically a cutoff of 2 is used (higher is better).
+2. Look at **replicate correlation**. Calculate the correlation between each replicate for each cell line (there are 3 replicates for each) in `LFC_TABLE.csv` and use only cell lines with reasonably good correlations across replicates. It is important to not that cell lines that are killed more tend to have higher replicate correlations so filtering in this way can remove more insensitive than sensitive cell lines.
+3. Look at the **$R^2$ and MSE of the dose-response curves** in `DRC_TABLE.csv` and filter for cell lines that have a good curve fit.
+4. Calculate the **monotonicity** of each cell line's dose-response curve. Filter out cell lines that do not respond monotonically (or nearly monotonically) to your compound.
diff --git a/info/MTS_pipeline.md b/info/MTS_pipeline.md
new file mode 100644
index 0000000..11e7f3e
--- /dev/null
+++ b/info/MTS_pipeline.md
@@ -0,0 +1,125 @@
+# MTS Processing Pipeline
+
+This document outlines the steps executed by each module of the MTS pipeline, with particular focus on inputs and outputs, that might be used by collaborators, from `LEVEL2` data through biomarker analysis.
+* [normalization](#normalization)
+* [qc](#qc)
+* [lfc](#lfc)
+* [batch_correction](#batch_correction)
+* [drc](#drc)
+* [biomarker](#biomarker)
+
+---
+
+## normalization
+
+#### Inputs
+* `cell_info`: metadata about cell lines/samples (rows)
+* `inst_info`: metadata about perturbations (columns)
+* `LEVEL2_MFI`: median fluorescence intensities (from Luminex)
+
+#### Outputs
+* `LEVEL3_LMFI`: normalized $\log_2(MFI)$ (called logMFI from now on)
+* `compound_key`: overview of the perturbations, projects, and doses
+
+#### Overview
+The normalization step $\log_2$ transforms the MFI values for each sample and then normalizes to the control barcodes (`pool_id == "CTLBC"`) in each well. Normalization is done by fitting a spline from the logMFI values to the median logMFI of the control barcode on the replicate. The resulting spline is then used to correct all data on the replicate.
+
+---
+
+## qc
+
+#### Inputs
+* `LEVEL3_LMFI`: normalized logMFI
+
+#### Outputs
+* `QC_TABLE`: quality control metrics for each cell line on each replicate
+
+#### Overview
+The QC step calculates a number of quality control metrics for each cell line on each replicate. Two of these metrics are used to filter cell lines out of downstream analysis:
+
+**Dynamic range:**
+$$DR = \mu_- - \mu_+$$
+where $\mu_-$ is the median logMFI in negative controls and $\mu_+$ is the median in positive controls. Cell lines with $DR > -\log_2{0.3} \approx 1.74$ pass QC on their replicate.
+
+**Error rate:**
+$$ER = \frac{FP + FN}{n}$$
+where $FP$ and $FN$ are the number of false positives and negatives of the omptimum threshold classifier between positive and negative controls. Cell lines with $ER <= 0.05$ pass QC on their replicate.
+
+In order for a cell line to be included in downstream analyses for a given plate, it must pass QC by the above metrics on more that 50% of replicates (only passing replicates are included).
+
+---
+
+## lfc
+
+#### Inputs
+* `LEVEL3_LMFI`: normalized logMFI
+* `QC_TABLE`: QC metrics for each cell line on each replicate
+
+#### Outputs
+* `LEVEL4_LFC`: log-fold change values for each cell line treatment on each replicate (that passes QC)
+* `LEVEL5_LFC`: replicate collapsed log-fold change values for each cell line treatment on each plate
+
+#### Overview
+The log-fold change step calculates the $\log_2$ fold-change (LFC) between treatment and negative controls on a given replicate.
+$$LFC = \log_2MFI_x - \mu_-$$
+where $\log_2MFI_x$ is the logMFI in treatment and $\log_2MFI_{\mu_-}$ is the median logMFI in the negative control.
+
+For those more familiar with viability, LFC is just log-viability such that
+$$v = 2^{LFC}$$
+
+The replicate collapsed LFC (`LEVEL5_LFC`) takes the median LFC for a cell line treatment across replicates of a given plate.
+
+---
+
+## batch_correction
+
+#### Inputs
+* `LEVEL4_LFC`: replicate level LFC values
+
+#### Outputs
+* `LEVEL4_LFC_COMBAT`: replicate level LFC values batch corrected using ComBat
+* `LEVEL5_LFC_COMBAT`: replicate collapsed batch corrected LFC values
+
+#### Overview
+The batch correction step corrects for batch effects by applying [ComBat](https://pubmed.ncbi.nlm.nih.gov/16632515/) to the `LEVEL4_LFC` data. `LEVEL5_LFC_COMBAT` takes the median ComBat corrected LFC for a cell line treatment across replicates of a given plate.
+
+---
+
+## drc
+
+**Note:** this module is meant to be run per compound per plate and therefore requires splitting into compound folders prior to execution (see [split](../split/README.md) module not covered in this document)
+
+#### Inputs
+* `LEVEL4_LFC_COMBAT` or `LEVEL4_LFC`: replicate level LFC data (ComBat corrected if available)
+
+#### Outputs
+* `DRC_TABLE`: dose-response curve parameters for each cell line in treatment
+
+#### Overview
+The dose-response curve step fits a robust four-parameter logistic curve to the response of each cell line of the form
+$$f(x) = b + \frac{a - b}{1 + e^{s \log \frac{x}{EC_{50}}}}$$
+where $x$ is the viability of the cell line at each dose. The parameters of the curve are
+* $b$: lower asymptote
+* $a$: upper asymptote
+* $s$: slope
+* $EC_{50}$: inflection point
+
+The upper asymptote is constrained to be near 1 and the lower asymptote is constrained to be between 0 and 1. In addition to these parameters, several others are reported in the table (see [column headers](./ColumnHeaders.md)), of particular note are $AUC$, the area under the dose response curve, and $IC_{50}$, the dose at which the curve reaches 50% viability.
+
+---
+
+## biomarker
+
+#### Inputs
+* `LEVEL5_LFC_COMBAT` or `LEVEL5_LFC`: replicate collapsed log-fold change data
+* `DRC_TABLE` (optional): dose-response parameters
+
+#### Outputs
+* `continuous_associations`: correlations with continuous CCLE and DepMap features
+* `discrete_associations`: lineage and mutation associations
+* `RF_table`: random forest feature level data
+* `model_table`: random forest model level results
+
+#### Overview
+
+The biomarker step generates univariate and multivariate associations with CCLE and DepMap data. The results are -omics features that are associated with response to treatment. For more specifics on the types of analyses see [here](./analysis_info.pdf). The datasets used for associations are listed in [`biomarker_files`](../biomarker_files/README.md).
diff --git a/info/changelog.md b/info/changelog.md
new file mode 100644
index 0000000..5397887
--- /dev/null
+++ b/info/changelog.md
@@ -0,0 +1,51 @@
+# Release Notes
+
+## MTS026
+***
+* Biomarker module pointing to consistent location for feature files (no changes to data)
+
+## CPS011
+***
+
+**Replicate correlation**
+
+Added functions to normalize that calculate 2 new metrics:
+ 1. The median value by detection plate+pool+well of the median difference in logMFI values for a given condition compared to the median of its replicates. If the absolute value of this difference is greater than 3, the plate/pool/well is flagged.
+ 2. The median value by detection plate/pool of the replicate correlations between each treatment condition and the median value across the replicates. If this correlation is <0.3, the plate/pool/well is flagged.
+
+If both conditions are met (ie, the correlation is <0.3 and the absolute value of the difference in median logMFI is > 3) then this data is removed. A file is also generated ending in *_POOL_WELLS_REMOVED.csv that indicates which values have been removed.
+
+* Removed unnamed columns (index) in QC_TABLE download
+
+
+## MTS025, CPS011, APS003
+***
+
+Added the floor range QC metric that looks at the separation between cell line signal (LMFI) in negative control and LMFI signal from a bead without its complementary target barcode present. Like with dynamic range, data from cell lines with separation < log2(0.3) are removed from the detection plate.
+
+
+## MTS024, CPS010, APS002
+***
+
+* Dose Response
+The dose response curve fitting module was revised substantially, motivated by examples where poor DRC fits were returned. Changes include:
+ 1. The module now attempts to fit the data points using a number of optimization methods and initializations using the dr4pl and drc packages. It then returns the fit with the lowest Mean Squared Error (MSE).
+ 2. The module constrains fits on single agent screens to have a slope parameter,s >0, to support a decreasing slope solution. This reflects the assumption that the assay aims to detect the reduction of cell viability by test agents. For combination screens, fits can have an increasing slope to detect antagonistic effects between agents.
+ 3. The Riemann AUC is also provided in the DRC table for each profile. In scenarios where a fit does not succeed, fit parameters are NA but the Riemann AUC is still provided.
+ 4. An error in the computation of R^2 of the dose-response curve fit has been corrected
+
+* Single-compound report and combination_solo report to use the same Rmd file
+
+* In rare occurrences where wells were reported skipped during the plate preparation, data for those wells are now filtered out
+* Lineages subabbreviations and abbreviations using OncoTree notation
+* Biomarkers updated to DepMap 23Q4 data
+
+* A value of 1 is now added to all mfi values in the raw_data matrix to prevent log(0)
+
+## EPS001
+***
+* Removed IC50 values from Extended Day Data Processing
+
+## MTS023, CPS009
+***
+* Biomarkers updated to DepMap 23Q2 data
diff --git a/info/dose_matching/Reprocessing_dose_handling.ipynb b/info/dose_matching/Reprocessing_dose_handling.ipynb
new file mode 100644
index 0000000..cb6a5eb
--- /dev/null
+++ b/info/dose_matching/Reprocessing_dose_handling.ipynb
@@ -0,0 +1,124 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import pandas as pd"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#read in instinfo\n",
+ "instinfo_original = pd.read_table('inst_original.txt')\n",
+ "instinfo_reprocessed = pd.read_table('inst_reprocessed.txt')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "\"\"\"\n",
+ "Function nearest dose from dose array\n",
+ "\n",
+ "dose: single float value\n",
+ "dose_array: array of floats\n",
+ "\"\"\"\n",
+ "def find_matching_dose(dose, dose_array):\n",
+ " #enforce typing\n",
+ " if type(dose)==str: \n",
+ " dose=float(dose)\n",
+ "\n",
+ " np_doses = np.array(dose_array.astype(float))\n",
+ " #subtract dose from all doses in dose_array\n",
+ " diffs = abs(dose - np_doses)\n",
+ " \n",
+ " #find index of min(abs(diffs)) and return index in dose_array\n",
+ " matched_dose = list(dose_array)[np.argmin(diffs)]\n",
+ " \n",
+ " return str(matched_dose)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#apply to Dataframe row-wise, feeding doses of reprocessed dataset\n",
+ "instinfo_original['matched_dose'] = instinfo_original.apply(\n",
+ " lambda row: find_matching_dose(row['pert_dose'], instinfo_reprocessed['pert_dose']),\n",
+ " axis=1\n",
+ ")\n",
+ "\n",
+ "instinfo_original['pert_idose'] = instinfo_original.apply(\n",
+ " lambda row: '{} {}'.format(row['matched_dose'], row['pert_dose_unit']),\n",
+ " axis=1\n",
+ ")\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "instinfo_original.head(5)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "instinfo_reprocessed.head(5)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/info/dose_matching/inst_original.txt b/info/dose_matching/inst_original.txt
new file mode 100644
index 0000000..8d79b86
--- /dev/null
+++ b/info/dose_matching/inst_original.txt
@@ -0,0 +1,25 @@
+profile_id pert_iname x_project_id pert_dose pert_idose pert_dose_unit pert_time
+PMTS037_PR300P_120H_X1_R2:B21 AZ-628 MTS016_VALIDATION_COMPOUND 0.123456790123 0.1234567901234568 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:C04 AZ-628 MTS016_VALIDATION_COMPOUND 0.00457247370828 0.004572473708276177 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:E24 AZ-628 MTS016_VALIDATION_COMPOUND 10.0 10.0 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:G08 AZ-628 MTS016_VALIDATION_COMPOUND 0.0411522633745 0.0411522633744856 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:I07 AZ-628 MTS016_VALIDATION_COMPOUND 1.11111111111 1.1111111111111112 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:K08 AZ-628 MTS016_VALIDATION_COMPOUND 3.33333333333 3.3333333333333335 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:M19 AZ-628 MTS016_VALIDATION_COMPOUND 0.37037037037 0.3703703703703704 uM uM 120H
+PMTS037_PR300P_120H_X1_R2:P04 AZ-628 MTS016_VALIDATION_COMPOUND 0.0137174211248 0.013717421124828532 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:E03 AZ-628 MTS016_VALIDATION_COMPOUND 0.123456790123 0.1234567901234568 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:E07 AZ-628 MTS016_VALIDATION_COMPOUND 10.0 10.0 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:G24 AZ-628 MTS016_VALIDATION_COMPOUND 3.33333333333 3.3333333333333335 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:J21 AZ-628 MTS016_VALIDATION_COMPOUND 0.0137174211248 0.013717421124828532 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:L16 AZ-628 MTS016_VALIDATION_COMPOUND 0.37037037037 0.3703703703703704 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:L17 AZ-628 MTS016_VALIDATION_COMPOUND 0.00457247370828 0.004572473708276177 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:L18 AZ-628 MTS016_VALIDATION_COMPOUND 0.0411522633745 0.0411522633744856 uM uM 120H
+PMTS037_PR300P_120H_X2_R2:P23 AZ-628 MTS016_VALIDATION_COMPOUND 1.11111111111 1.1111111111111112 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:C13 AZ-628 MTS016_VALIDATION_COMPOUND 3.33333333333 3.3333333333333335 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:C23 AZ-628 MTS016_VALIDATION_COMPOUND 0.00457247370828 0.004572473708276177 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:G04 AZ-628 MTS016_VALIDATION_COMPOUND 0.0137174211248 0.013717421124828532 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:I12 AZ-628 MTS016_VALIDATION_COMPOUND 0.123456790123 0.1234567901234568 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:K09 AZ-628 MTS016_VALIDATION_COMPOUND 1.11111111111 1.1111111111111112 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:K24 AZ-628 MTS016_VALIDATION_COMPOUND 0.37037037037 0.3703703703703704 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:L17 AZ-628 MTS016_VALIDATION_COMPOUND 0.0411522633745 0.0411522633744856 uM uM 120H
+PMTS037_PR300P_120H_X3_R2:N12 AZ-628 MTS016_VALIDATION_COMPOUND 10.0 10.0 uM uM 120H
diff --git a/info/dose_matching/inst_reprocessed.txt b/info/dose_matching/inst_reprocessed.txt
new file mode 100644
index 0000000..ecc8050
--- /dev/null
+++ b/info/dose_matching/inst_reprocessed.txt
@@ -0,0 +1,25 @@
+profile_id pert_iname x_project_id pert_dose pert_idose pert_dose_unit pert_time
+PMTS037_PR300P_120H_X1.L2_R3:B21 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.1235 0.1235 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:C04 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.004572 0.004572 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:E24 AZ-628 MTS016_VALIDATION_COMPOUNDS 10.0 10.0 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:G08 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.04115 0.04115 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:I07 AZ-628 MTS016_VALIDATION_COMPOUNDS 1.111 1.111 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:K08 AZ-628 MTS016_VALIDATION_COMPOUNDS 3.333 3.333 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:M19 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.3704 0.3704 uM uM 120H
+PMTS037_PR300P_120H_X1.L2_R3:P04 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.01372 0.01372 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:E03 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.1235 0.1235 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:E07 AZ-628 MTS016_VALIDATION_COMPOUNDS 10.0 10.0 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:G24 AZ-628 MTS016_VALIDATION_COMPOUNDS 3.333 3.333 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:J21 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.01372 0.01372 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:L16 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.3704 0.3704 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:L17 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.004572 0.004572 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:L18 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.04115 0.04115 uM uM 120H
+PMTS037_PR300P_120H_X2.L2_R3:P23 AZ-628 MTS016_VALIDATION_COMPOUNDS 1.111 1.111 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:C13 AZ-628 MTS016_VALIDATION_COMPOUNDS 3.333 3.333 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:C23 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.004572 0.004572 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:G04 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.01372 0.01372 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:I12 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.1235 0.1235 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:K09 AZ-628 MTS016_VALIDATION_COMPOUNDS 1.111 1.111 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:K24 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.3704 0.3704 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:L17 AZ-628 MTS016_VALIDATION_COMPOUNDS 0.04115 0.04115 uM uM 120H
+PMTS037_PR300P_120H_X3.L2_R3:N12 AZ-628 MTS016_VALIDATION_COMPOUNDS 10.0 10.0 uM uM 120H
diff --git a/info/images/biomarkers.png b/info/images/biomarkers.png
new file mode 100644
index 0000000..d2eeeb1
Binary files /dev/null and b/info/images/biomarkers.png differ
diff --git a/info/images/nutlin.png b/info/images/nutlin.png
new file mode 100644
index 0000000..bc44336
Binary files /dev/null and b/info/images/nutlin.png differ
diff --git a/info/images/validation.png b/info/images/validation.png
new file mode 100644
index 0000000..0659e8a
Binary files /dev/null and b/info/images/validation.png differ
diff --git a/info/preamble.tex b/info/preamble.tex
new file mode 100644
index 0000000..92bd004
--- /dev/null
+++ b/info/preamble.tex
@@ -0,0 +1,8 @@
+\usepackage{float}
+\let\origfigure\figure
+\let\endorigfigure\endfigure
+\renewenvironment{figure}[1][2] {
+ \expandafter\origfigure\expandafter[H]
+} {
+ \endorigfigure
+}
diff --git a/jenkinsfile b/jenkinsfile
new file mode 100644
index 0000000..a835f84
--- /dev/null
+++ b/jenkinsfile
@@ -0,0 +1,82 @@
+import groovy.json.JsonSlurper
+
+REPO="prismcmap"
+SRC_TAG="latest"
+DEST_TAG="production"
+
+def DOCKER_NAMES=[
+ "assemble-db",
+ "batch-correct-module",
+ "biomarker-module",
+ "collate",
+ "csv2json",
+ "deal",
+ "depmap_versions",
+ "drc-module",
+ "extract-biomarker",
+ "landing",
+ "lfc-module",
+ "merge-csvs",
+ "norm-module",
+ "pivot",
+ "pivot_splits",
+ "register-mts",
+ "reports",
+ "qc-screen",
+ "split",
+ "stack"
+]
+
+
+def getCheckedOutGitCommitHash() {
+ def gitFolder = "$projectDir/.git/"
+ def takeFromHash = 12
+ /*
+ * '.git/HEAD' contains either
+ * in case of detached head: the currently checked out commit hash
+ * otherwise: a reference to a file containing the current commit hash
+ */
+ def head = new File(gitFolder + "HEAD").text.split(":") // .git/HEAD
+ def isCommit = head.length == 1 // e5a7c79edabbf7dd39888442df081b1c9d8e88fd
+ // def isRef = head.length > 1 // ref: refs/heads/master
+
+ if(isCommit) return head[0].trim().take(takeFromHash) // e5a7c79edabb
+
+ def refHead = new File(gitFolder + head[1].trim()) // .git/refs/heads/master
+ refHead.text.trim().take takeFromHash
+}
+def parallelStagesMap = DOCKER_NAMES.collectEntries {
+ ["${it}" : generateStage(it,getCheckedOutGitCommitHash())]
+}
+def generateStage(job,hash) {
+ return {
+ stage("stage: ${job}") {
+ sh script: "docker pull ${REPO}/${job}:${SRC_TAG}"
+ sh script: "docker tag $REPO/${job}:${SRC_TAG} ${REPO}/${job}:${DEST_TAG} ${REPO}/${job}:${hash}"
+ sh script: "docker push ${REPO}/${job}:${DEST_TAG}"
+ }
+ }
+}
+
+
+pipeline {
+ agent any
+ environment {
+ DOCKERHUB_CREDENTIALS= credentials('dockerhub-cred-jasiedu')
+ }
+ stages {
+ stage('Login to Docker Hub') {
+ steps{
+ sh 'echo $DOCKERHUB_CREDENTIALS_PSW | sudo docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
+ echo 'Login Completed'
+ }
+ }
+ stage('parallel stage') {
+ steps {
+ script {
+ parallel parallelStagesMap
+ }
+ }
+ }
+ }
+}
diff --git a/jenkinsfile-build-pipeline-docker b/jenkinsfile-build-pipeline-docker
new file mode 100644
index 0000000..f581b97
--- /dev/null
+++ b/jenkinsfile-build-pipeline-docker
@@ -0,0 +1,153 @@
+/**
+* Copy of the jenkins job that creates and tags dockers for dev
+* Tags are develop
+*/
+
+import groovy.json.JsonSlurper
+
+REPO="prismcmap"
+DEV_TAG="develop"
+WORKSPACE_DIR="/ci-data/jenkins/workspace/BUILD-PIPELINE-DOCKER/"
+GIT_FOLDER = "/ci-data/jenkins/workspace/START-BUILD-PIPELINE/.git/"
+URL_PREFIX="https://api.clue.io/api/pipeline_versions/upsertWithWhere?where=";
+
+def getCheckedOutGitCommitHash() {
+
+ def takeFromHash = 12
+ /*
+ * '.git/HEAD' contains either
+ * in case of detached head: the currently checked out commit hash
+ * otherwise: a reference to a file containing the current commit hash
+ */
+ def head = new File("${GIT_FOLDER}" + "HEAD").text.split(":") // .git/HEAD
+ def isCommit = head.length == 1 // e5a7c79edabbf7dd39888442df081b1c9d8e88fd
+ // def isRef = head.length > 1 // ref: refs/heads/master
+
+ if(isCommit) return head[0].trim().take(takeFromHash) // e5a7c79edabb
+
+ def refHead = new File("${GIT_FOLDER}" + head[1].trim()) // .git/refs/heads/master
+ refHead.text.trim().take takeFromHash
+}
+
+def parallelStagesMap(DIRS,GIT_PREVIOUS_COMMIT,GIT_COMMIT){
+ def mp = [:]
+ CHANGED = sh(returnStdout: true, script: "git diff --name-only ${GIT_PREVIOUS_COMMIT} ${GIT_COMMIT}").trim()
+
+ if(CHANGED){
+ CHANGED_LINES = CHANGED.split("\n")
+ if(CHANGED_LINES.size() > 0){
+ for( String CHANGED_LINE : CHANGED_LINES ){
+ try{
+ def (DIR)=CHANGED_LINE.tokenize( '/' );
+ API_JOB = DIR.replace("/","") + "_API"
+
+ DOCKER_FILE_EXISTS=sh(returnStdout: true, script: "test -f ${DIR}/Dockerfile && echo 'FILE exists.'").trim()
+
+ if(DOCKER_FILE_EXISTS){
+
+ GIT_COMMIT_HASH = getCheckedOutGitCommitHash()
+ mp.put("${DIR}",generateStage(DIR,GIT_COMMIT_HASH))
+ mp.put("${API_JOB}",generateAPIStage(API_JOB,DIR,GIT_COMMIT_HASH))
+ }
+
+ }
+ catch(err){
+ println(err);
+ }
+ }
+ }
+ }
+ return mp
+}
+def generateAPIStage(job,repo_name,docker_hash) {
+ return {
+ stage("stage: ${job}") {
+ String name = REPO + "/" + repo_name;
+ String where = '{"name":"' + name + '","docker_tag":"' + DEV_TAG + '"}';
+ String whereClause = java.net.URLEncoder.encode(where, "UTF-8");
+ def message = '{"name":"' + name + '","docker_tag":"' + DEV_TAG + '","docker_hash": "' + docker_hash + '"}';
+ def post = new URL(URL_PREFIX + whereClause).openConnection();
+ post.setRequestMethod("POST")
+ post.setDoOutput(true)
+ post.setRequestProperty("Content-Type", "application/json")
+ post.setRequestProperty("user_key",API_KEY)
+ post.getOutputStream().write(message.getBytes("UTF-8"));
+ def postRC = post.getResponseCode();
+ println(postRC);
+ }
+ }
+}
+def generateStage(job,HASH_TAG) {
+ return {
+ stage("stage: ${job}") {
+ dir("${job}") {
+ sh script: "pwd"
+ sh script: "docker build -t ${REPO}/${job}:${DEV_TAG} -t ${REPO}/${job}:${HASH_TAG} --rm=true ."
+ sh script: "docker push ${REPO}/${job}:${DEV_TAG}"
+ sh script: "docker push ${REPO}/${job}:${HASH_TAG}"
+ }
+ }
+ }
+}
+pipeline{
+ agent any
+ environment {
+ DOCKERHUB_CREDENTIALS= credentials('dockerhub-cred-jasiedu')
+ GIT_PREVIOUS_COMMIT=''
+ GIT_COMMIT=''
+ API_KEY = credentials('API_KEY')
+ }
+ stages {
+ stage('Login to Docker Hub') {
+ steps{
+ sh 'echo $DOCKERHUB_CREDENTIALS_PSW | sudo docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
+ echo 'Login Completed'
+ }
+ }
+ stage("Git Checkout"){
+ steps{
+ git credentialsId: '4c3bd279-468f-4837-a095-3366964ae937', url: 'https://github.com/cmap/dockerized_mts.git' ,branch: 'dev'
+ echo 'Git Checkout Completed'
+ }
+ }
+ stage('Configure') {
+ steps {
+ script {
+ GIT_PREVIOUS_COMMIT = sh(returnStdout: true, script: "git rev-parse --short 'HEAD^'").trim()
+ GIT_COMMIT=sh(returnStdout: true, script: "git rev-parse --short HEAD").trim()
+ }
+ }
+ }
+ stage('Checkout code') {
+ steps {
+ script {
+ DIRS=sh(returnStdout: true, script: "ls -d -- */").trim()
+ DIR_PATHS = DIRS.split("\n")
+ mp = parallelStagesMap(DIR_PATHS,GIT_PREVIOUS_COMMIT,GIT_COMMIT)
+ mp.each { key, value ->
+ println("Key: ${key}, Value: ${value}")
+ }
+
+ if(mp.size() > 0){
+ parallel mp
+ }
+ }
+ }
+ }
+ }
+ post {
+ always {
+ cleanWs()
+ dir("${env.WORKSPACE}@tmp") {
+ deleteDir()
+ }
+ dir("${env.WORKSPACE}@script") {
+ deleteDir()
+ }
+ dir("${env.WORKSPACE}@script@tmp") {
+ deleteDir()
+ }
+ }
+ }
+}
+
diff --git a/jenkinsfile-release-pipeline-dockers b/jenkinsfile-release-pipeline-dockers
new file mode 100644
index 0000000..fabb1ad
--- /dev/null
+++ b/jenkinsfile-release-pipeline-dockers
@@ -0,0 +1,123 @@
+/**
+* Copy of the jenkins job that creates and tags dockers for production
+* Tags are production, latest and the short git hash
+*/
+
+import groovy.json.JsonSlurper
+
+REPO="prismcmap"
+PROD_TAG="production"
+LATEST_TAG="latest"
+GIT_FOLDER = "/ci-data/jenkins/workspace/RELEASE-PIPELINE-DOCKERS/.git/"
+URL_PREFIX="https://api.clue.io/api/pipeline_versions/upsertWithWhere?where=";
+
+def getCheckedOutGitCommitHash() {
+
+ def takeFromHash = 12
+ /*
+ * '.git/HEAD' contains either
+ * in case of detached head: the currently checked out commit hash
+ * otherwise: a reference to a file containing the current commit hash
+ */
+ def head = new File("${GIT_FOLDER}" + "HEAD").text.split(":") // .git/HEAD
+ def isCommit = head.length == 1 // e5a7c79edabbf7dd39888442df081b1c9d8e88fd
+ // def isRef = head.length > 1 // ref: refs/heads/master
+
+ if(isCommit) return head[0].trim().take(takeFromHash) // e5a7c79edabb
+
+ def refHead = new File("${GIT_FOLDER}" + head[1].trim()) // .git/refs/heads/master
+ refHead.text.trim().take takeFromHash
+}
+
+def parallelStagesMap(DIRS){
+ GIT_COMMIT_HASH = getCheckedOutGitCommitHash()
+ def mp = [:]
+ for( String DIR : DIRS ){
+ DIR = DIR.replace("/","")
+ if(!DIR.equals("docker_base" ) && !DIR.equals("biomarker_files" ) && !DIR.equals("info" )){
+
+ DOCKER_FILE_EXISTS=sh(returnStdout: true, script: "test -f ${DIR}/Dockerfile && echo FILE exists.").trim()
+ if(DOCKER_FILE_EXISTS){
+ API_JOB = DIR + "_API"
+ mp.put("${DIR}",generateStage(DIR,GIT_COMMIT_HASH))
+ mp.put("${API_JOB}",generateAPIStage(API_JOB,DIR,GIT_COMMIT_HASH))
+ }
+ }
+ }
+ return mp
+}
+def generateAPIStage(job,repo_name,docker_hash) {
+
+ return {
+ stage("stage: ${job}") {
+ String name = REPO + "/" + repo_name;
+ String where = '{"name":"' + name + '","docker_tag":"' + PROD_TAG + '"}';
+ String whereClause = java.net.URLEncoder.encode(where, "UTF-8");
+ def message = '{"name":"' + name + '","docker_tag":"' + PROD_TAG + '","docker_hash": "' + docker_hash + '"}';
+ def post = new URL(URL_PREFIX + whereClause).openConnection();
+ post.setRequestMethod("POST")
+ post.setDoOutput(true)
+ post.setRequestProperty("Content-Type", "application/json")
+ post.setRequestProperty("user_key", API_KEY)
+ post.getOutputStream().write(message.getBytes("UTF-8"));
+ def postRC = post.getResponseCode();
+ println(postRC);
+ }
+ }
+}
+def generateStage(job,HASH_TAG) {
+ return {
+ stage("stage: ${job}") {
+ dir("${job}") {
+ sh script: "docker build -t ${REPO}/${job}:${LATEST_TAG} -t ${REPO}/${job}:${PROD_TAG} -t ${REPO}/${job}:${HASH_TAG} --rm=true ."
+ sh script: "docker push ${REPO}/${job}:${LATEST_TAG}"
+ sh script: "docker push ${REPO}/${job}:${PROD_TAG}"
+ sh script: "docker push ${REPO}/${job}:${HASH_TAG}"
+ }
+ }
+ }
+}
+pipeline{
+ agent any
+ environment {
+ DOCKERHUB_CREDENTIALS= credentials('dockerhub-cred-jasiedu')
+ API_KEY = credentials('API_KEY')
+ }
+ stages {
+ stage('Login to Docker Hub') {
+ steps{
+ sh 'echo $DOCKERHUB_CREDENTIALS_PSW | sudo docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
+ echo 'Login Completed'
+ }
+ }
+ stage("Git Checkout"){
+ steps{
+ git credentialsId: '4c3bd279-468f-4837-a095-3366964ae937', url: 'https://github.com/cmap/dockerized_mts.git' ,branch: 'master'
+ echo 'Git Checkout Completed'
+ }
+ }
+ stage('Build Dockers') {
+ steps {
+ script {
+ DIRS=sh(returnStdout: true, script: "ls -d -- */").trim()
+ DIR_PATHS = DIRS.split("\n")
+ parallel parallelStagesMap(DIR_PATHS)
+ }
+ }
+ }
+ }
+ post {
+ always {
+ cleanWs()
+ dir("${env.WORKSPACE}@tmp") {
+ deleteDir()
+ }
+ dir("${env.WORKSPACE}@script") {
+ deleteDir()
+ }
+ dir("${env.WORKSPACE}@script@tmp") {
+ deleteDir()
+ }
+ }
+ }
+}
diff --git a/landing_pages/.dockerignore b/landing_pages/.dockerignore
new file mode 100644
index 0000000..f9ec605
--- /dev/null
+++ b/landing_pages/.dockerignore
@@ -0,0 +1,4 @@
+README.md
+build_docker.sh
+docker_push_tags.sh
+docker_run.sh
diff --git a/landing_pages/Dockerfile b/landing_pages/Dockerfile
new file mode 100644
index 0000000..f37bdf3
--- /dev/null
+++ b/landing_pages/Dockerfile
@@ -0,0 +1,23 @@
+FROM rocker/verse:4.0.4
+
+MAINTAINER Andrew Boghossian
+LABEL clue.mts.pipeline.clue.io.version="0.0.1"
+LABEL clue.mts.pipeline.clue.io.vendor="PRISM"
+
+RUN apt-get update && apt-get install -y jq
+COPY ./src/install_packages.R /src/install_packages.R
+
+RUN mkdir -p /clue/bin && Rscript /src/install_packages.R
+
+COPY ./rmarkdown/ /rmarkdown/
+COPY ./render_reports.R /render_reports.R
+
+WORKDIR /
+
+COPY ./aws_batch.sh /clue/bin/aws_batch
+
+RUN ["chmod","-R", "+x", "/clue/bin"]
+ENV PATH /clue/bin:$PATH
+ENTRYPOINT ["aws_batch"]
+
+CMD ["-help"]
diff --git a/landing_pages/README.md b/landing_pages/README.md
new file mode 100644
index 0000000..c451873
--- /dev/null
+++ b/landing_pages/README.md
@@ -0,0 +1,70 @@
+# Landing Page Module
+
+**For internal use**
+
+This module creates the landing pages for each project with links to downloads and reports.
+
+## Landing page overview
+
+Landing pages are generated with RMarkdown (files in [`rmarkdown`](./rmarkdown)).
+
+## Docker image
+
+To install the docker image from [Docker Hub](https://hub.docker.com/repository/docker/prismcmap/landing) run:
+
+```
+docker pull prismcmap/landing:latest
+```
+
+To get a specific version replace `latest` with the version desired.
+
+## Execution
+
+The entrypoint for the Docker is `aws_batch.sh` which is a wrapper around `render_reports.R` which calls the rmarkdown files. For more information using R run
+
+```
+Rscript render_reports.R --help
+usage: render_reports.R [-h] [-d DATA_DIR] [-o OUT_DIR] [-p PROJECT_NAME]
+ [-l VAL_LINK]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -d DATA_DIR, --data_dir DATA_DIR
+ Input directory (project)
+ -o OUT_DIR, --out_dir OUT_DIR
+ Output directory. Default is working directory.
+ -p PROJECT_NAME, --project_name PROJECT_NAME
+ Project folder name
+ -l VAL_LINK, --val_link VAL_LINK
+ Link to validation compound landing page
+```
+
+### Example usage with R
+
+R execution requires installing the correct R packages which can be installed by running:
+
+```
+Rscript src/install_packages.R
+```
+
+Then the module can be run;
+
+```
+Rscript render_reports.R -d ~/Desktop/clue_data/project -o ~/Desktop/mts_results/project -p project -l https://analysis.clue.io/validation_compunds/index.html
+```
+
+### Example usage with Docker
+
+Docker execution requires mounting directories with the `-v` option in order to obtain results.
+
+```
+docker run \
+ -it \
+ -v ~/Desktop/clue_data/project:/in_data \
+ -v ~/Desktop/mts_results/project:/out_data \
+ prismcmap/landing:latest \
+ -d /in_data \
+ -o /out_data \
+ -p project \
+ -l https://analysis.clue.io/validation_compunds/index.html
+```
diff --git a/landing_pages/aws_batch.sh b/landing_pages/aws_batch.sh
new file mode 100644
index 0000000..910b228
--- /dev/null
+++ b/landing_pages/aws_batch.sh
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+# read in flagged arguments
+while test $# -gt 0; do
+ case "$1" in
+ -f| --compound_key_path)
+ shift
+ projects=$1
+ ;;
+ -d| --data_dir)
+ shift
+ data_dir=$1
+ ;;
+ -o| --out)
+ shift
+ out_dir=$1
+ ;;
+ -p| --project_name)
+ shift
+ project_name=$1
+ ;;
+ -c| --combination)
+ shift
+ combination=$1
+ ;;
+ -b| --build_name)
+ shift
+ build_name=$1
+ ;;
+ -l| --val_link)
+ shift
+ val_link=$1
+ ;;
+ -qc| --no_mts_qc)
+ shift
+ no_mts_qc=$1
+ ;;
+ -bc| --no_batch_correct)
+ shift
+ no_batch_correct=$1
+ ;;
+ *)
+ printf "Unknown parameter: %s \n" "$1"
+ shift
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+combination=${combination:-0}
+echo $combination
+
+no_mts_qc=${no_mts_qc:-0}
+echo no_mts_qc
+
+no_batch_correct=${no_batch_correct:-0}
+echo no_batch_correct
+
+batch_index=0
+if [[ ! -z $projects ]]
+then
+ if [[ ! -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
+ batch_index=${AWS_BATCH_JOB_ARRAY_INDEX}
+ fi
+ project=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].x_project_id')
+ combination=$(cat "${projects}" | jq -r --argjson index ${batch_index} '.[$index].combination_project')
+ data_dir="${data_dir}"/"${project,,}"/"${project^^}"
+ val_link=$(find "${out_dir}" -type d -name "*validation_compounds" -execdir basename '{}' ';')
+ val_link=../"${val_link}"/index.html
+ out_dir="${out_dir}"/"${project,,}"
+ project_name="${project^^}"
+fi
+
+echo "${data_dir}" "${out_dir}" "${project_name}" "${val_link}"
+
+if [[ ! -d $out_dir ]]
+then
+ mkdir -p $out_dir
+fi
+
+args=(
+ -d "${data_dir}"
+ -o "${out_dir}"
+ -p "${project_name}"
+ -b "${build_name}"
+ -l "${val_link}"
+ -c "${combination}"
+ -qc "${no_mts_qc}"
+ -bc "${no_batch_correct}"
+)
+
+echo Rscript /render_reports.R "${args[@]}"
+Rscript /render_reports.R "${args[@]}"
+
+exit_code=$?
+
+echo "$exit_code"
+exit $exit_code
diff --git a/landing_pages/docker_build_and_push.sh b/landing_pages/docker_build_and_push.sh
new file mode 100755
index 0000000..eb6430f
--- /dev/null
+++ b/landing_pages/docker_build_and_push.sh
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+REPO="prismcmap"
+NAME="landing_pages"
+
+#VERSION="v0.2.2" #change the version number for each new build
+TAG="testing"
+
+#change the version number for each new build
+docker build --platform linux/amd64 -t $REPO/$NAME:$TAG --rm=true .
+
+#docker push $REPO/$NAME:$VERSION
+docker push $REPO/$NAME:$TAG
+
+# triggers
\ No newline at end of file
diff --git a/landing_pages/docker_run.sh b/landing_pages/docker_run.sh
new file mode 100755
index 0000000..f34a7f3
--- /dev/null
+++ b/landing_pages/docker_run.sh
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+docker run --rm \
+--name bar \
+-v ~/WebstormProjects/PMTS018/projects/:/data \
+-it prismcmap/landing \
+-d /data/mts018_dmc_loxo/MTS018_DMC_LOXO \
+-o /data/mts018_dmc_loxo \
+-p MTS018_DMC_LOXO
diff --git a/landing_pages/render_reports.R b/landing_pages/render_reports.R
new file mode 100644
index 0000000..50f24c2
--- /dev/null
+++ b/landing_pages/render_reports.R
@@ -0,0 +1,51 @@
+library(argparse)
+
+parser <- ArgumentParser()
+# specify our desired options
+parser$add_argument("-d", "--data_dir", default="", help="Input directory (project)")
+parser$add_argument("-o", "--out_dir", default="", help = "Output directory. Default is working directory.")
+parser$add_argument("-p", "--project_name", default="", help = "Project folder name")
+parser$add_argument("-b", "--build_name", default="", help = "Build name")
+parser$add_argument("-l", "--val_link", default="", help = "Link to validation compound landing page")
+parser$add_argument("-c", "--combination_project", default="0", help = "Project has combination files")
+parser$add_argument("-qc", "--no_mts_qc", default="0", help = "Flag to not run MTS report QC code chunk")
+parser$add_argument("-bc", "--no_batch_correct", default="0", help = "Flag indicating there was no batch correction")
+
+# get command line options, if help option encountered print help and exit
+args <- parser$parse_args()
+
+if (as.numeric(args$combination_project) == 0) {
+ combination_project = FALSE
+} else if (as.numeric(args$combination_project) == 1) {
+ combination_project = TRUE
+} else {
+ print("Invalid combination argument. Must be 1 or 0")
+}
+
+if (as.numeric(args$no_mts_qc) == 0) {
+ run_mts_qc = TRUE
+} else if (as.numeric(args$no_mts_qc) == 1) {
+ run_mts_qc = FALSE
+} else {
+ print("Invalid MTS QC flag argument. Must be 1 or 0")
+}
+
+if (as.numeric(args$no_batch_correct) == 0) {
+ batch_correct = TRUE
+} else if (as.numeric(args$no_batch_correct) == 1) {
+ batch_correct = FALSE
+} else {
+ print("Invalid batch correction argument. Must be 1 or 0")
+}
+
+
+rmarkdown::render("rmarkdown/landing_page.Rmd",
+ output_file = "index.html",
+ output_dir = args$out_dir,
+ params = list(data_dir=args$data_dir,
+ project_name=args$project_name,
+ build_name=args$build_name,
+ val_link=args$val_link,
+ combination_project=combination_project,
+ run_mts_qc=run_mts_qc,
+ batch_correct=batch_correct))
diff --git a/landing_pages/rmarkdown/depmap_scripts.html b/landing_pages/rmarkdown/depmap_scripts.html
new file mode 100644
index 0000000..998bb5c
--- /dev/null
+++ b/landing_pages/rmarkdown/depmap_scripts.html
@@ -0,0 +1,15 @@
+
diff --git a/landing_pages/rmarkdown/landing_page.Rmd b/landing_pages/rmarkdown/landing_page.Rmd
new file mode 100644
index 0000000..617e271
--- /dev/null
+++ b/landing_pages/rmarkdown/landing_page.Rmd
@@ -0,0 +1,393 @@
+---
+params:
+ data_dir: /Users/shiker/Documents/trouble_shoot/aps001/aps001_validation_compounds/APS001_VALIDATION_COMPOUNDS
+ project_name: APS001_VALIDATION_COMPOUNDS
+ build_name: PAPS
+ val_link: ~/Desktop/cps005_validation_compounds/CPS005_VALIDATION_COMPOUNDS
+ combination_project: TRUE
+ run_mts_qc: TRUE
+ batch_correct: TRUE
+title: "`r params$project` Overview"
+date: "`r format(Sys.time(), '%d %B, %Y')`"
+output:
+ html_document:
+ toc: false
+ theme: paper
+ highlight: kate
+ includes:
+ after_body: depmap_scripts.html
+---
+
+```{r setup, include=FALSE, include=FALSE, echo=FALSE, message=FALSE, warning=FALSE}
+knitr::opts_chunk$set(include=T, echo=FALSE, message=FALSE, warning=FALSE)
+library(DT)
+library(tidyverse)
+library(ggthemes)
+library(magrittr)
+library(cowplot)
+
+
+
+# get paths to data files
+data_files <- list.files(file.path(params$data_dir, "data"))
+data_filesp <- paste0("./", data_files)
+data_files <- paste0("./", params$project_name,"/","data","/",data_files)
+proj_length <- length(strsplit(params$project_name, split = "_", fixed = T)[[1]])
+names(data_files) <- stringr::word(data_filesp, proj_length + 1, -2, sep = fixed("_"))
+
+if (params$batch_correct) {
+file_text <- c(
+ "QC" = "QC metrics",
+ "LEVEL3_LMFI" = "Raw and normalized fluorescence values",
+ "LEVEL4_LFC_COMBAT" = "Viability data",
+ "LEVEL5_LFC_COMBAT" = "Replicate-collapsed viability data",
+ "DRC" = "Dose response curve data"
+)
+} else {
+ file_text <- c(
+ "QC" = "QC metrics",
+ "LEVEL3_NORMALIZED_COUNTS" = "Normalized counts",
+ "LEVEL4_LFC" = "Viability data",
+ "LEVEL5_LFC" = "Replicate-collapsed viability data",
+ "DRC" = "Dose response curve data"
+)
+}
+
+biomarker_text <- c(
+ "continuous" = "Continuous associations",
+ "discrete" = "Discrete associations",
+ "model" = "Random forest model statistics",
+ "RF" = "Random forest features"
+)
+
+matrix_text <- c(
+ "LFC" = "Log-fold change matrix",
+ "AUC" = "AUC matrix",
+ "IC50" = "IC50 matrix"
+)
+
+combination_text <- c(
+ "synergy" = "Synergy Table",
+ "bliss_mss" = "Bliss MSS Table"
+)
+
+# get compound info
+
+inst_info <- data.table::fread(list.files(file.path(params$data_dir, "data"), pattern = "inst_info", full.names = T))
+
+inst <- inst_info %>% dplyr::distinct(pert_iname, pert_id)
+
+# gathering positive control
+
+
+screen <- substr(inst_info$screen %>% unique(), start = 1, stop = 3)
+pos_control <- ifelse(screen == "APS", "Puromycin", "Bortezomib")
+
+
+neg_control <- inst_info$pert_vehicle %>% unique()
+
+name_map <- inst$pert_iname
+names(name_map) <- stringr::str_replace_all(inst$pert_id, pattern = fixed("|"), "_") %>% toupper()
+
+# get paths to html files
+plates <- list.dirs(params$data_dir, recursive = F)
+report_list <- list()
+print(plates)
+for (i in 1:length(plates)) {
+ print(plates[[i]])
+ plate <- stringr::word(plates[[i]], -1, sep = "/")
+ print(plate)
+ if (plate == "data") next
+ report_files <- list.files(plates[[i]], pattern = "_report.html", recursive = T)
+ print(report_files)
+ if (length(report_files) > 0) {
+ report_list[[plate]] <- paste0("./", params$project_name, "/", plate, "/", report_files)
+ }
+}
+
+drc_list <- list()
+for (i in 1:length(plates)) {
+
+ plate <- stringr::word(plates[[i]], -1, sep = "/")
+ if (plate == "data") next
+ drc_files <- list.files(plates[[i]], pattern = "_drc.html", recursive = T)
+ if (length(drc_files) > 0) {
+ drc_list[[plate]] <- paste0("./", params$project_name, "/", plate, "/", drc_files)
+ }
+}
+
+# push NAs to bottom in tables
+options(DT.TOJSON_ARGS = list(na = "string"))
+```
+
+
+
+
+
+
+
+
+
+## Screen Info {.tabset .tabset-fade .tabset-pills}
+
+### Deliverables
+
+1. Mean fluorescent intensities (MFI) for each control and treatment well. This is a measure of the abundance of each PRISM barcode for each replicate/treatment/cell line, available in long-table format: `Compound Data > Raw fluorescence values` and `Normalized fluorescence values`.
+2. QC table containing all the dynamic range and error rate statistics, available: `Compound Data > QC metrics`.
+3. Normalized (log-fold change) cell-viability values with respect to negative control (`r neg_control`), available in long-table format (`Compound Data > Viability data`) and (`Compound Data > Replicate collapsed viability data`) for median-collapsed values across cell lines.
+4. Dose-response parameters for the compound, available: `Compound Data > Dose-response curve data`, and dose-response curves for each treatment/cell line in `Dose-Response Curves`. **Please note:** we are sometimes unable to fit a dose response curve for some cell lines so they will not be included in this file.
+5. Biomarker and correlation analysis using omics and genetic dependency data from [depmap.org](https://depmap.org/portal/download/all/) (DepMap Public 23Q2). Biomarkers are identified by selecting the genomic features that best explain the observed profile of sensitivity in the PRISM cell line panel. Univariate analysis for continuous and discrete variables at `Biomarker Data > Continuous associations` and `Discrete associations` respectively. Multivariate analysis available at `Biomarker Data > Random forest features` and with model level statistics at `Biomarker Data > Random forest model statistics`.
+
+Explanations of the columns within each file are available on [GitHub](https://github.com/cmap/dockerized_mts/blob/master/info/ColumnHeaders.md)
+
+
+### Data Processing
+
+I. For each plate, we first normalize the logMFI ($\log_2$ mean fluorescence intensity) of the `r neg_control` wells to their median logMFI.
+
+II. Each detection well contains 10 control barcodes in increasing abundances as spike-in controls. A monotonic smooth p-spline is fit for each control barcode detection well to normalize the abundance of each barcode to the corresponding value in the plate-wise median `r neg_control` profiles. Next, all the logMFI values in the well are transformed through the inferred spline function to correct for amplification and detection artifacts.
+
+III. Next, the separability between negative and positive control treatments is assessed. In particular, we use the error rate of the optimum simple threshold classifier between the control samples for each cell line and plate combination. Error rate is a measure of overlap of the two control sets and is defined as $$ER = \frac{FP + FN}{n}$$ where $FP$ is false positives, $FN$ is false negatives, and $n$ is the total number of controls. A threshold is set between the distributions of positive and negative control logMFI values (with everything below the threshold said to be positive and above said to be negative) such that this value is minimized. Additionally, we filter based on the dynamic range of each cell line. Dynamic range is defined as $$DR = \mu_{-} - \mu_{+}$$ where $\mu_{+/-}$ stand for the median of the normalized logMFI values in positive/negative control samples.
+
+III. We filter out cell lines with error rate above $0.05$ and a dynamic range less than $-log2(0.3) \approx 1.74$ from the downstream analysis. Any cell line that has less than 2 passing replicates is also omitted for the sake of reproducibility. Finally, we compute viability by normalizing with respect to the median negative control for each plate. Log-fold-changes are computed as $$\text{log-viability} = \log_2\left({x} \right) - \log_2\left({\mu_{-}} \right)$$
+where $\log_2\left({x} \right)$ is the corrected logMFI value in the treatment and $\log_2\left(\mu_{-} \right)$ is the median corrected logMFI in the negative control wells in the same plate.
+
+IV. Log-viability scores are corrected for batch effects coming from pools and culture conditions using the `ComBat` algorithm as described in [Johnson et al.](https://www.ncbi.nlm.nih.gov/pubmed/16632515).
+
+V. We fit a robust four-parameter logistic curve to the response of each cell line to the compound: $$f(x) = b + \frac{a-b}{1+e^{s\log{\frac{x}{EC50}}}}$$ and compute AUC and IC50 values for each dose-response curve.
+
+VI. Finally, the replicates are collapsed to a treatment level profile by computing the median score for each cell line.
+
+
+### Points of contact
+
+For any questions please see our [FAQs](https://github.com/cmap/dockerized_mts/blob/master/info/FAQs.md) or reach out to prism@broadinstitute.org.
+
+All code used to process this data, as well as further information about the data processing and FAQs, is available on [GitHub](https://github.com/cmap/dockerized_mts).
+
+```{r validation, results='asis'}
+if (stringr::str_detect(params$project_name, coll("validation", ignore_case = T), negate = T)) {
+ cat("\n\n## Validation Compound Data \n\n")
+ cat(paste0("Data from validation compounds run in this screen can be found [here](", params$val_link, ").\n\n"))
+}
+```
+
+## Reports and Downloads {.tabset .tabset-pills}
+```{r}
+ download_files = data.frame(
+ title=character(0),
+ type=character(0),
+ url=character(0),
+ project=character(0),
+ build=character(0)
+ ) #instantiate
+```
+
+### Report Files
+
+These are the links to the HTML reports for each compound submitted to the screen. A summary of the analyzed data is included in the report for each compound, with a cell line viability heatmap, interactive plots, and tables. Please note that this report includes both fit-curve (AUC and IC50) and individual dose data.
+
+```{r reports, results='asis'}
+if (length(report_list) > 0) {
+ for (i in 1:length(report_list)) {
+ cat("\n\n", names(report_list)[[i]], "\n\n")
+ for (j in 1:length(report_list[[i]])) {
+ file_name <- stringr::word(report_list[[i]][[j]], -1, sep = "/") %>% stringr::word(., 1, -2, sep = "_")
+ if (stringr::word(file_name, -1, sep = "_") == "combination") {
+ comp <- paste(name_map[stringr::word(file_name, 1, -2, sep = "_")], "combination")
+ } else {
+ comp <- name_map[file_name]
+ }
+ cat("* [", comp,
+ "](", report_list[[i]][[j]], ") \n")
+ }
+ }
+} else {
+ cat("No report files found for this project.")
+}
+```
+
+### Dose-Response Curves
+
+These are links to interactive HTML files of the dose-response figures for each compound. Note: some cell lines that do not exhibit any sensitivity during the course of the assay cannot be fit with a dose response curve, so there may be a reduced number of cell lines represented.
+
+```{r drcs, results='asis'}
+if (length(drc_list) > 0) {
+ for (i in 1:length(drc_list)) {
+ cat("\n\n", names(drc_list)[[i]], "\n\n")
+ for (j in 1:length(drc_list[[i]])) {
+ file_name <- stringr::word(drc_list[[i]][[j]], -1, sep = "/") %>% stringr::word(., 1, -2, sep = "_")
+ comp <- name_map[file_name]
+ cat("* [", comp,
+ "](", drc_list[[i]][[j]], ") \n")
+ }
+ }
+} else {
+ cat("No dose-response files found for this project.")
+}
+```
+
+### Compound Data
+
+These are the links to files containing QC and sensitivity data for all compounds. Includes the QC and sensitivity data for all compounds.
+
+```{r data, results='asis'}
+for (i in 1:length(data_files)) {
+ if (names(data_files)[i] %in% names(file_text)) {
+ cat("* [", file_text[names(data_files)[i]], "](", data_files[i], ") \n")
+
+ download_files %<>% add_row(
+ title=file_text[names(data_files)[i]],
+ type="compound_data",
+ url=file.path(str_to_lower(params$project_name), str_replace(data_files[i], "./", "")),
+ project=params$project_name,
+ build=params$build_name
+ )
+
+ } else {
+ next
+ }
+}
+```
+
+### Biomarker Data
+
+These are links to files containing biomarker analysis results for all compounds. Includes tables for univariate and multivariate (Random Forest model) analysis.
+
+```{r biomarkers, results='asis'}
+for (i in 1:length(data_files)) {
+ if (names(data_files)[i] %in% names(biomarker_text)) {
+ cat("* [", biomarker_text[names(data_files)[i]], "](", data_files[i], ") \n")
+
+ download_files %<>% add_row(
+ title=biomarker_text[names(data_files)[i]],
+ type="biomarker_data",
+ url=file.path(str_to_lower(params$project_name), str_replace(data_files[i], "./", "")),
+ project=params$project_name,
+ build=params$build_name
+ )
+
+ } else {
+ next
+ }
+}
+```
+
+### Matrix versions
+
+These are links to files that are matrices of data for all compounds. These files are compatible with the [DepMap data explorer tool](https://depmap.org/portal/interactive/).
+
+```{r matrices, results='asis'}
+for (i in 1:length(data_files)) {
+ if (names(data_files)[i] %in% names(matrix_text)) {
+ cat("* [", matrix_text[names(data_files)[i]], "](", data_files[i], ") \n")
+ cat('\n',sep="")
+
+ download_files %<>% add_row(
+ title=matrix_text[names(data_files)[i]],
+ type="depmap",
+ url=file.path(str_to_lower(params$project_name), str_replace(data_files[i], "./", "")),
+ project=params$project_name,
+ build=params$build_name
+ )
+
+ } else {
+ next
+ }
+}
+
+#add row with location for zip file
+download_files %<>% add_row(
+ title="Project Downloads",
+ type="all",
+ url=file.path(str_to_lower(params$project_name), params$project_name, "data", paste0(params$project_name, ".zip")),
+ project=params$project_name,
+ build=params$build_name
+ )
+
+write.csv(download_files, file.path(params$data_dir, "data", "downloads.csv"), row.names=FALSE)
+```
+
+```{r combinations, eval=params$combination_project}
+knitr::asis_output("### Combination Downloads\\n")
+knitr::asis_output("\nThese are links to files that are matrices of data for all combination compounds \n")
+```
+
+```{r comb_files, results='asis', eval=params$combination_project}
+for (i in 1:length(data_files)) {
+ if (names(data_files)[i] %in% names(combination_text)) {
+ cat("* [", combination_text[names(data_files)[i]], "](", data_files[i], ") \n")
+
+ download_files %<>% add_row(
+ title=combination_text[names(data_files)[i]],
+ type="combination_data",
+ url=file.path(str_to_lower(params$project_name), str_replace(data_files[i], "./", "")),
+ project=params$project_name,
+ build=params$build_name
+ )
+ } else {
+ next
+ }
+}
+```
+
+```{r}
+#add row with location for zip file
+download_files %<>% add_row(
+ title="Project Downloads",
+ type="all",
+ url=file.path(str_to_lower(params$project_name), params$project_name, "data", paste0(params$project_name, ".zip")),
+ project=params$project_name,
+ build=params$build_name
+ )
+
+write.csv(download_files, file.path(params$data_dir, "data", "downloads.csv"), row.names=FALSE)
+```
+
+
+## QC Information
+
+For each cell line screened we compute the error rate, which is based on the overlap between positive and negative control distributions and dynamic range, which is the difference between positive and negative control medians (see Data Processing section for more details). These values are plotted below on the left with cell line replicate pairs that pass QC shown in blue and failing pairs in red. On the right the percentage and number of passing lines is shown for each plate relevant to this project. The complete list of these statistics, including other measures of spread, is tabulated and presented below.
+
+```{r qc, fig.width=12,eval=params$run_mts_qc}
+# aesthetics
+theme_set(theme_bw())
+dodger = position_dodge(width = 0.9)
+
+# QC plot
+qc_file <- list.files(file.path(params$data_dir, "data"), "QC_TABLE", full.names = T)
+qc_table <- data.table::fread(qc_file)
+
+p1 <- ggplot(qc_table) +
+ geom_point(aes(x = dr, y = error_rate, color = pass)) +
+ geom_vline(aes(xintercept = -log2(0.3)), lty = 2, lwd =.5) +
+ geom_hline(aes(yintercept = 0.05), lty = 2, lwd =.5) +
+ facet_wrap(pert_plate + pert_time ~ .) +
+ labs(x = "Dynamic range", y = "Error rate", color = "Pass") +
+ scale_color_fivethirtyeight(limits = c("TRUE", "FALSE")) +
+ theme(legend.position = "none")
+p2 <
+ 
+
+
+
+
+
+
+
+
+
+