# ---
# title: "fadnUtils"
# author: ""
# output: #rmarkdown::html_vignette
#   word_document
# # vignette: >
# #   %\VignetteIndexEntry{fadnUtils}
# #   %\VignetteEngine{knitr::rmarkdown}
# #   %\VignetteEncoding{UTF-8}
# ---

# fadnUtils

The fadnUtils package facilitates the efficient handling of FADN data within the R language framework. Furthermore, the package is targeted for use within the JRC D.4 context. This means that there is a specific temporal pattern of how a user interacts with the package (see Figure plot).
![plot](inst/examples/pic/workflow.png)

More specifically, after a request for FADN data from DG-AGRI, this data is delivered to JRC D.4 in csv format. 

# Installation 

You can install the development version from Gitlab with:

```{r}
devtools::install_git("https://git-dmz.thuenen.de/mindstep/fadnutilspackages")
```

Then the Related R packages can be installed.

```{r}

requiredPackages = c('fadnUtils','data.table', 'devtools','jsonlite', 'ggplot2')
for(p in requiredPackages){
  if(!require(p,character.only = TRUE)) install.packages(p)
  library(p,character.only = TRUE)
}
```

# Usage in Brief
After loaded the packages, you will have a functinal R package on your computer. Then, we will talk about using your package online.

1. Create a working directory
    - a user-defined data directory
1. Import CSV FADN data
    - convert the csv data into raw r-data
    - convert raw r-data into str r-data
1. Load r-data and structured r-data
1.  Perform analysis

## 1. Create a working directory
Frist, User sets a working directory. Make sure the relative path stays within `CurrentProjectDirectory`. 
```{r}
# using a local directory
CurrentProjectDirectory = "D:/public/yang/MIND_STEP/New_test_fadnUtils"
create.data.dir(folder.path = CurrentProjectDirectory)
set.data.dir(CurrentProjectDirectory)
get.data.dir()
```
### Required files

We request FADN data from DG-AGRI, which is delivered to us in csv format. In order to work efficiently with R, we should convert the csv-data to an r friendly format, this step is done with help of a human-readable file, called `raw_str_map.file`. Both files are necessary. `inst/examples` is the folder for use cases that contain fadnUtils package examples and json files.

1. FADN data in csv format: the data for loading
2. A json file for extracting the variables

### Folder Structure 
A working directory is specified arbitrarily by the user. This structure helps data management and maintenance. The directory looks like this:

```base
CurrentProjectDirectory/
+-- csv
+-- fadnUtils.metadata.json
+-- rds
\-- spool
    \-- readme.txt
```
* csv: CSV files are stored here
* fadnUtils.metadata.json: containing the mapping from the fadn.raw.rds to the fadn.str.rds data
* rds: placing r-data in the "rds" directory
* spool: keeping related files

## 2. Import CSV FADN data
First, we will import the data into an R-friendly format using the fadnUtils package.

### Convert the csv data into raw r-data
The raw data will be added to a `rds` directory. We use a convenient function from this package to convert the csv file into raw r-data.

```{r}

fadn.data.dir <- "D:/public/data/fadn/lieferung_20210414/csv/"

# load data for country BEL and year 2009

convert.to.fadn.raw.rds(

      file.path =  paste0(fadn.data.dir, "BEL2009.csv"),

      sepS = ",",

      fadn.country = "BEL",

      fadn.year = 2009
      #keep.csv = T # copy csv file in csv.dir

      )

```
At any time, we can check for the current data dir, what csv files (countries, year) are loaded.
```{r}
show.data.dir.contents()
```

### Convert raw r-data into structured r-data
Then, We convert raw data into structured data. Broadly, there are 3 steps to including data in an R package: 

1. setting a structured data in the `structured` directory, 
2. checking the `raw_str_map.file` that all variables can be converted.
3. converting the structured data successfully into `structured` directory.

#### Set a `structured` directory for saving the structured data
We set a `test` folder to placing the structured data.

```{r}
rds.dir = paste0(get.data.dir(),"/rds/")
# set a structured name for for saving the structured r-data in rds.dir
new.str.name = "test"
# set a extraction_dir
dir.create(paste0(rds.dir, new.str.name))
new.extraction.dir = paste0(rds.dir, new.str.name)
```

#### Check the variables in the `raw_str_map.file`
 Before conversion it is recommended to use `check.column()` method, ensuring that all variables in the`raw_str_map.file` can be converted.
```{r results='hide', message=FALSE, warning=FALSE}

list_vars = check.column(
              # a rds file or a csv file

              importfilepath = paste0(rds.dir, "fadn.raw.2009.BEL.rds"),

              # a json file
              jsonfile = "D:/public/yang/MIND_STEP/2014_after_copy.json",
              # write a new json file without unmatched variables
              rewrite_json = T,
              # save the new json in extraction_dir
              extraction_dir = new.extraction.dir)

```


#### Convert the raw data into structured r-data using the checked json file
Finally, We can convert a raw r-data to str r-data using a external json file. For more details on converting in fadnUtils packages, `see USE_CASE.R`.
```{r}

convert.to.fadn.str.rds(fadn.country = "BEL",
                        fadn.year = 2009,
                        str.name = new.str.name # extraction_dir
)

convert.to.fadn.str.rds(fadn.country = "BEL",
                        fadn.year = 2009,
                        raw_str_map.file = "D:/public/yang/MIND_STEP/new_sample/test01/raw_str_map.json", # a external json file
                        str.name = new.str.name, # extraction_dir
                        force_external_raw_str_map = T,
                        DEBUG = F
                        )