# hic-te

A Nextflow workflow to analyze HiC data from SRA (NCBI Short Read Archive) for 3D contacts between repeat families. Slightly biased (but not limited to) towards LTR retroTEs and plant genomes.

*SYNOPSIS*

nextflow run [FILEBASE].nf -profile LIST[,LIST...] [PARAMS...]

To run the pipeline, the following parameters are mandatory:

*DATA*  

**reads**  
Reads from a Hi-C experiment. The easiest way to provide this is by listing their SRA id.  
--sra_run SRR14458670  

**reference**  
Reference genome corresponding to the organism the reads belong to. The reference should be in the fasta format.  
--reference Osativa_323_v7.0.fasta

**plantsat**  
This file contains annotations of the satellite DNA in plants.  
--plantsat PlantSat.fa

**exons_gff3**  
This file contains the annotations of the exons in the chosen reference genome.  
--exons_gff3 Osativa_323_v7.0.gene_exons.gff3

*SOFTWARE*  

Please provide paths to the local installations or use docker/singularity containers as implemented in the workflow.  

BBTools, https://jgi.doe.gov/data-and-tools/bbtools/  
--bbmap_java_path bbmap.jar
Provide path to the directory containing bbmap.jar  
--bbmap_reformat_path reformat.sh
Provide path to the function reformat.sh that is part of the bbmap package  

RepeatExplorer, http://repeatexplorer.org/  
Provide path to the clustering function of the Repeat Explorer. This is important to provide reference-free annotations of repeats in the Hi-C data.  
--re_seqclust_path repex_tarean/seqclust

*RUNNING THE PIPELINE*  

Typical commands for running the pipeline are as follows:  

**test data run on the local machine**  
nextflow run main_TE_2.nf -profile test --skip_align false --data_dir "output"  

**test data run with the support of docker/singularity**  
nextflow run main_TE_2.nf -profile test,docker"  
nextflow run main_TE_2.nf -profile test,singularity"

*OPTIONAL PARAMETERS*

**alternative input to save resources**

    skip_nester             =   true
    input_nester            =   "/mnt/nas/biodata/hic-te/data2/data"

    skip_bow_index          =   false
    input_bow_index         =   "/mnt/nas/biodata/hic-te/ref_Slyc"

    skip_diachromatic       =   false
    input_diachromatic      =   "${params.data_dir}"

    skip_align              =   false
    input_align_bam         =   "/mnt/nas/biodata/hic-te/output/${params.sra_run}_diachromatic_truncated_paired_uniq_ref.bam"

    skip_re                 =   false
    input_re                =   "/mnt/nas/biodata/xnguyen3/hic-te/input/RE_output"

    skip_re_contigs_map     =   false
    input_re_contigs_gff    =   "/mnt/nas/biodata/hic-te/output/annotations/Osativa_323_v7.0_contigs.annotated_filtered.gff"

**general**

    help                    =   false

    data_dir                =   "/mnt/nas/biodata/hic-te/output"

    report_dir              =   "${params.data_dir}/pipeline_report"

    save_intermediate       =   true

**TE-greedy-nester settings**

    gff_suffix              =   "_genome_browser"

**diachromatic**

    REnzyme                 =   "DpnII"

**sra**

    skip_prefetch_sra       =   false

    skip_convert_fastq      =   false

    fastq_dir               =   "${params.data_dir}/${params.sra_run}"

**annotations**

    exons_gff3              =   "/mnt/nas/biodata/hic-te/Slycopersicum_691_ITAG4.0.gene_exons.gff3"

    plantsat                =   "/mnt/nas/biodata/hic-te/PlantSat_Solanum.fa"

    mRNA                    =   "/mnt/nas/biodata/hic-te/Slycopersicum_691_ITAG4.0.mRNA.gff3"

    rm                      =   "/mnt/nas/biodata/hic-te/Slycopersicum_691_ITAG4.0.repeatmasked_assembly_SL4.0.gff3"

**repeat explorer**

    re_seqclust_path        =   "/home/lexa/git/repex_tarean/seqclust" // full path to seqclust

    clustering_threshold    =   0.01

    RE_run                  =   "RE_output_" + params.sra_run + "_" + params.clustering_threshold

*NORMALIZATION METHODS*

After counting all valid HiC pairs in the pipeline a table is created that contains family names in two columns (family1, family2) and in cases based on the reference genome also mapped positions (pos1, pos2). The number of combinations observed between positions and repeat families contains technical and methodological biases. For example there are many more pairs observed for adjacent positions on the same chromosomes compared to long-distance or interchromosomal HiC pairs. Some kind of normalization is therefore necessary before reporting basic statistics or creating heatmap visualizations. Choosing the right normalization method is far from trivial. After careful consideration, we chose three different methods that we use in parallel in towards the end of calculations in the pipeline when a familyxfamily matrix underlying each heatmap is calculated.

**joint probability**

The baseline probability of observing a HiC pair between family A and family B is estimated as joint probability of individual probabilities for observing family A or family B in a given HiC read.

  p(A,B) = p(A).p(B). 

All counts of A-B pairs are divided by this number.

**label permutation**

Family names in the table in columns family1 and family2 are randomly assigned to random (and therefore possibly different) rows of the table. A matrix is also created from this altered table. All counts of A-B pairs are divided by corresponding values in this matrix. 

**annotation shuffling**

While creating the HiC pair table, a parallel table is made, which uses chromosomal positions randomly shuffled along the reference genome. The pair count matrix constructed from such table is used for normalization as above (see label permutation). 

*NOTES*

**adding genomes/organisms**

While the pipeline will happily run on any HiC data and the corresponding reference genome, there are some limitation when running the vanilla gitlab version in such manner. Repeat classification done by Repeat Explorer, TE-greedy-nester and inner blast annotation scripts is plant-oriented, using the Neumann et al. classification scheme. TE-greedy nester enriches the  annotations for LTR retrotransposons. Alternative repeat annotations may be prefered for other organisms. Tandem repeats are collected as input in the PlantSat(other_annotations) file. This file is mapped onto the reference genome, so any sequences can be added in FASTA format.

**envisioned modifications**

  + adding tandem repeats to PlantSat.fa
  + changing classification scheme

**reusing intermediate data for speed-up**

The pipeline contains a number of intermediate steps that will be repeated if another analysis is run for the same organism or the same downloaded SRR file. See #alternative input to save resources# above (under optional parameters). For example once the reference genome is analyzed by TE-greedy-nester and an index is created by bowtie2, these steps can be skipped in subsequent runs against the same organism (reference genome). Just set skip_nester and skip_bow_index to 'true'. The same goes for Repeat Explorer output (set skip-re to 'true'). In each case the path or file with pre-calculated data needs to be provided by a corresponding input_* parameter.
