# Functional tests: Usage

Each step of metagwgs produces a series of files. We want to be able to determine if the modifications we perform on metagwgs have an impact on any of these files (presence, contents, format, ...). You'll find more info about how the files are tested at the end of this page.

## I. Launch functional tests

### On genologin

Procedure to launch functional tests on the genologin cluster with access to `/work/project/plateforme/metaG/functional_test`:

1. Install metagwgs:

```bash
git clone https://forgemia.inra.fr/genotoul-bioinfo/metagwgs.git
export METAG_PATH="./metagwgs"
```
2. Launch metagWGS: 
```bash 
sbatch $METAG_PATH/functional_tests/launch_example.sh
```
3. Launch functional tests:

```bash
module load system/Python-3.7.4
python $METAG_PATH/functional_tests/main.py -step 08_binning -exp_dir /work/project/plateforme/metaG/functional_test/metagwgs-test-datasets/small/output -obs_dir ./results
```
Don't forget to modify the `METAG_PATH` variable, if metagwgs is not in your working directory.  

This procedure allows you to perform functional tests for small data (Short reads), to test HiFi data you need to include the parameters `--input "/work/project/plateforme/metaG/functional_test/metagwgs-test-datasets/hifi/input/samplesheet.csv" --type "HiFi" ` to the launch_example.sh script. And change the expected directory for the python command. 

If you do not have access to genologin or the directory, you need to download the data and banks, please follow the procedure below:

### Donwload of DB and data

1. Install metagwgs as described here: [installation doc](../docs/installation.md)
2. Get datasets: three datasets are currently available for these functional tests at https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets.git

```bash
git clone git@forgemia.inra.fr:genotoul-bioinfo/metagwgs-test-datasets.git

or

wget https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets.git
```
3. Get data banks: download [this archive](http://genoweb.toulouse.inra.fr/~choede/FT_banks.tgz) and decompress its contents in any folder. 

```bash
wget http://genoweb.toulouse.inra.fr/~choede/FT_banks.tgz
tar -xvzf FT_banks.tgz
```

This archive contains data banks for:
    - **Kaiju** (_kaijudb_refseq_2020-05-25_)
    - **Diamond** (_refseq_bacteria_2021-05-20_)
    - **NCBI Taxonomy** (_taxonomy_2021-12-7_ )
    - **Eggnog Mapper** (_eggnog-mapper-2.1.9_)
    - **gtdb** (release207_v2)
    - **Homo_sapiens.GRCh38_chr21** for host genome
    - **uniref100.KO.1.dmnd** for checkm2


4. Set enviroment variables and adapt configuration file

```bash
export METAG_PATH="/path/to/metagwgs"
export DATASET="/path/to/metagwgs-dataset-tests"
export DATABANK="/path/to/FT_banks"

sed -i "s,\/work\/project\/plateforme\/metaG\/functional_test\/FT_banks,$DATABANK,g" $METAG_PATH/conf/functional_test.config
sed -i "s,\/work\/project\/plateforme\/metaG\/functional_test\/metagwgs-dataset-tests,$DATASET,g" $METAG_PATH/conf/functional_test.config
sed -i "s,\/work\/project\/plateforme\/metaG\/functional_test\/singularity_img,$METAG_PATH\/env" $METAG_PATH/conf/functional_test.config
```

5. Launch metagWGS

```bash 
sed -i "s,\$DATASET,$DATASET,g" $DATASET/small/input/samplesheet.csv
 
sbatch $METAG_PATH/functional_tests/launch_example.sh
```
6. Launch functional tests

```bash
module load system/Python-3.7.4
python $METAG_PATH/functional_tests/main.py -step 07_taxo_affi -exp_dir $DATASET/small/output -obs_dir ./results
```

> To run locally, delete the following lines from the file `$METAG_PATH/conf./functional_test.config` before launch metagwgs:
> ```
> process.executor = 'slurm'
> singularity.runOptions = "-B /work/bank/ -B /bank -B /work2 -B / work -B /save -B /home -B /work/project -B /usr/local/bioinfo"
> process.queue = 'workq' 
> ```

This procedure allows you to perform functional tests for small data (Short reads), to test HiFi data you need to include the parameters `--input "/work/project/plateforme/metaG/functional_test/metagwgs-test-datasets/hifi/input/samplesheet.csv" --type "HiFi" ` to the launch_example.sh script. And change the expected directory for the python command. 

## III. Test skips and check processes  (WARNING: not up-to-date, needs to be updated, will be replaced by dry-run)

The script `test_parameters_and_processes.py` check if execution with parameters specified in `expected_processes.tsv`, run processes as expected.

To use it :
1. Retrieve databank and datasets (small) as describe in functional test above. (not necessary if on genologin with acces to ... )
2. Create command file: 
  ``` 
  cut -f 1 $METAG_PATH/functional_tests/expected_processes/expected_processes_sr.tsv  | tail -n +2 > ./cmd_sr.sh
  ``` 
3. Launch on the cluster the commands:
  ``` 
  sarray cmd_sr.sh
  ```
4. Launch `test_parameters_and_processes.py`
  ```
  module load bioinfo/Nextflow-v21.04.1
  $METAG_PATH/functional_tests/test_parameters_and_processes.py --file $METAG_PATH/functional_tests/expected_processes/expected_processes_sr.tsv 
  ```

This procedure allows you to perform functional tests for small data (Short reads), to test HiFi data you need to replace every `expected_processes_sr.tsv` by `expected_processes_sr_HiFi.tsv`. 

>**NOTE: more information on the command used to produce each dataset in [small](https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets/-/tree/small) and [mag](https://forgemia.inra.fr/genotoul-bioinfo/metagwgs-test-datasets/-/tree/mag) READMEs**

## IV. Output

A ft_\[step\].log file is created for each step of metagwgs. It contains information about each test performed on given files.

Exemple with ft_01_clean_qc.log:

```
Expected directory: metagwgs-test-datasets/output/01_clean_qc
vs
Observed directory: results/01_clean_qc

------------------------------------------------------------------------------

File:           01_1_cleaned_reads/cleaned_c_R1.fastq.gz
Test method:    zdiff
Test result:    Passed

------------------------------------------------------------------------------

File:           01_1_cleaned_reads/cleaned_c_R2.fastq.gz
Test method:    zdiff
Test result:    Passed


...


=========================================
-----------------------------------------

Testing the 01_clean_qc step of metagWGS:

Total:      36
Passed:     36 (100.0%)
Missed:     0 (0.0%)
Not tested: 0

-----------------------------------------
=========================================
```

If a test resulted in 'Failed' instead of 'Passed', the stdout is printed in log.

Sometimes, files are not tested because present in exp_dir but not in obs_dir. Then a log ft_\[step\].not_tested is created containing names of missing files. In 02_assembly, there are two possible assembly programs that can be used: metaspades and megahit, resulting in this .not_tested log file. Not tested files are not counted in missed count.


### Test methods

5 simple test methods are used:

sort_diff: simple bash difference between two files
`diff <(sort exp_path) <(sort obs_path)`

zdiff: simple bash difference between two gzipped files
`zdiff exp_path obs_path`

no_header_diff: remove the headers of .annotations and .seed_orthologs files
`diff <(grep -w "^?#" exp_path) <(grep -w "^?#" obs_path)`

cut_diff: exception for cutadapt.log file
`diff <(tail -n+6 exp_path) <(tail -n+6 obs_path)`

not_empty: in python, check if file is empty
`test = path.getsize(obs_path) > 0`
