<!-- README.md is generated from README.Rmd. Please edit that file -->

# fadnUtils

Develop by Dimitrios Kremmydas (JRC) and Xinxin Yang (THÜNEN)

The fadnUtils package facilitates the efficient handling of FADN data within the R language framework. Furthermore, the package is targeted for use within the JRC D.4 context. This means that there is a specific temporal pattern of how a user interacts with the package (see Figure plot). ![plot](inst/examples/pic/workflow.png)

More specifically, after a request for FADN data from DG-AGRI, this data is delivered to JRC D.4 in csv format.

# Installation

You can install the development version from Thuenen or IIASA Gitlab with:

``` r
# Thuenen gitlab
devtools::install_git("https://git-dmz.thuenen.de/mindstep/fadnutilspackages", force = TRUE)
# IIASA gitlab
devtools::install_git("https://gitlab.iiasa.ac.at/mind-step/fadnutilspackage")

```

Then the Related R packages can be installed.

``` r

requiredPackages = c('fadnUtils','data.table', 'devtools','jsonlite', 'ggplot2')
for(p in requiredPackages){
  if(!require(p,character.only = TRUE)) install.packages(p)
  library(p,character.only = TRUE)
}
```

    ## Loading required package: fadnUtils

    ## fadnUtils is loaded.

    ## Loading required package: data.table

    ## Loading required package: devtools

    ## Loading required package: usethis

    ## Loading required package: jsonlite

    ## Loading required package: ggplot2

# Usage in Brief

After loaded the packages, you will have a functional R package on your computer. Then, we will talk about using your package online.

1.  Create a working directory
    -   a user-defined data directory
2.  Import CSV FADN data
    -   convert the csv data into raw r-data
    -   convert raw r-data into str r-data
3.  Load r-data and structured r-data
4.  Perform analysis

## 1. Create a working directory

Frist, User sets a working directory. Make sure the relative path stays within `CurrentProjectDirectory`.

``` r

# using a local directory
CurrentProjectDirectory = "D:/public/yang/MIND_STEP/New_test_fadnUtils"
create.data.dir(folder.path = CurrentProjectDirectory)

```

    ## This is already a data.dir structure. Doing nothing.

``` r

set.data.dir(CurrentProjectDirectory)
get.data.dir()
```

    ## [1] "D:/public/yang/MIND_STEP/New_test_fadnUtils"

### Required files

We request FADN data from DG-AGRI, which is delivered to us in csv format. In order to work efficiently with R, we should convert the csv-data to an r friendly format, this step is done with help of a human-readable file, called `raw_str_map.file`. Both files are necessary. `inst/examples` is the folder for use cases that contain fadnUtils package examples and json files.

1.  FADN data in csv format: the data for loading
2.  A json file for extracting the variables

### Folder Structure

A working directory is specified arbitrarily by the user. This structure helps data management and maintenance. The directory looks like this:

``` base

CurrentProjectDirectory/
+-- csv
+-- fadnUtils.metadata.json
+-- rds
\-- spool
    \-- readme.txt
```

-   csv: CSV files are stored here
-   fadnUtils.metadata.json: containing the mapping from the fadn.raw.rds to the fadn.str.rds data
-   rds: placing r-data in the "rds" directory
-   spool: keeping related files

## 2. Import CSV FADN data

First, we will import the data into an R-friendly format using the fadnUtils package.

### Convert the csv data into raw r-data

The raw data will be added to a `rds` directory. We use a convenient function from this package to convert the csv file into raw r-data.

``` r

fadn.data.dir <- "D:/public/data/fadn/lieferung_20210414/csv/"

# load data for country BEL and year 2009

convert.to.fadn.raw.rds(

      file.path =  paste0(fadn.data.dir, "BEL2009.csv"),

      sepS = ",",

      fadn.country = "BEL",

      fadn.year = 2009
      #keep.csv = T # copy csv file in csv.dir

      )

```

    ## File D:/public/data/fadn/lieferung_20210414/csv/BEL2009.csv does not exist. Exiting ...

    ## [1] FALSE

At any time, we can check for the current data dir, what csv files (countries, year) are loaded.

``` r

show.data.dir.contents()
```

### Convert raw r-data into structured r-data

Then, We convert raw data into structured data. Broadly, there are 3 steps to including data in an R package:

1.  setting a structured data in the `structured` directory,
2.  checking the `raw_str_map.file` that all variables can be converted.
3.  converting the structured data successfully into `structured` directory.

#### Set a `structured` directory for saving the structured data

We set a `test` folder to placing the structured data.

``` r

rds.dir = paste0(get.data.dir(),"/rds/")
# set a structured name for for saving the structured r-data in rds.dir
new.str.name = "test"
# set a extraction_dir
dir.create(paste0(rds.dir, new.str.name))
new.extraction.dir = paste0(rds.dir, new.str.name)
```

#### Check the variables in the `raw_str_map.file`

Before conversion it is recommended to use `check.column()` method, ensuring that all variables in the`raw_str_map.file` can be converted.

``` r

list_vars = check.column(
              # a rds file or a csv file

              importfilepath = paste0(rds.dir, "fadn.raw.2009.BEL.rds"),

              # a json file
              jsonfile = "D:/public/yang/MIND_STEP/2014_after_copy.json",
              # write a new json file without unmatched variables
              rewrite_json = T,
              # save the new json in extraction_dir
              extraction_dir = new.extraction.dir)

```

#### Convert the raw data into structured r-data using the checked json file

Finally, We can convert a raw r-data to str r-data using a external json file. For more details on converting in fadnUtils packages, `see USE_CASE.R`.

``` r
convert.to.fadn.str.rds(fadn.country = "BEL",
                        fadn.year = 2009,
                        str.name = new.str.name # extraction_dir
)

convert.to.fadn.str.rds(fadn.country = "BEL",
                        fadn.year = 2009,
                        raw_str_map.file = "D:/public/yang/MIND_STEP/new_sample/test01/raw_str_map.json", # a external json file
                        str.name = new.str.name, # extraction_dir
                        force_external_raw_str_map = T,
                        DEBUG = F
                        )
```

#### Files Structure in `rds` folder

After conversion, we can see the `rds` folder:

-   `fadn.raw.2009.BEL.rds`: raw r-data for country "BEL" and year "2009"
-   `test`: extraction\_dir for saving the structured r-data and extracting json file
-   `fadn.str.2009.BEL.rds`: structured s-data for for country of "BEL" and year of "2009"
-   `raw_str_map.json`: default json file
-   `rewrite_2014_after_copy.json`: modified json file after checking the variables

``` base