<!-- README.md is generated from README.Rmd. Please edit that file -->

# fadnUtils

Develop by Dimitrios Kremmydas (JRC) and Xinxin Yang (THÜNEN)

The fadnUtils package facilitates the efficient handling of FADN data
within the R language framework. Furthermore, the package is targeted
for use within the JRC D.4 context. This means that there is a specific
temporal pattern of how a user interacts with the package (see Figure
plot). ![plot](inst/examples/pic/workflow.png)

More specifically, after a request for FADN data from DG-AGRI, this data
is delivered to JRC D.4 in csv format.

# Installation

You can install the development version from Thuenen or IIASA Gitlab
with:

``` r
# Thuenen gitlab
devtools::install_git("https://git-dmz.thuenen.de/mindstep/fadnutilspackages", force = TRUE)
# IIASA gitlab
devtools::install_git("https://gitlab.iiasa.ac.at/mind-step/fadnutilspackage")

```

Then the Related R packages can be installed.

``` r

library(fadnUtils)
```

    ## fadnUtils is loaded.

``` r
requiredPackages = c('data.table', 'devtools','jsonlite', 'ggplot2')

for(p in requiredPackages){
  if(!require(p,character.only = TRUE)) install.packages(p)
  library(p,character.only = TRUE)
}
```

    ## Loading required package: data.table

    ## Loading required package: devtools

    ## Loading required package: usethis

    ## Loading required package: jsonlite

    ## Loading required package: ggplot2

# Usage in Brief

After loaded the packages, you will have a functional R package on your
computer. Then, we will talk about using your package online.

1.  Create a working directory

    - a user-defined data directory

2.  Import CSV FADN data

    - convert the csv data into raw r-data
    - convert raw r-data into str r-data

3.  Load r-data and structured r-data
4.  Perform analysis

5.  Translate between various NUTS Version (FADN Region, NUTS1, NUTS2,
    NUTS2)

## 1. Create a working directory

Working directory is a path that sets the location of any files your
save from R. User sets a working directory. Make sure the relative path
stays within `CurrentProjectDirectory`.

``` r

# using a local directory
CurrentProjectDirectory = "D:/public/yang/MIND_STEP/New_test_fadnUtils"
create.data.dir(folder.path = CurrentProjectDirectory)

```

    ## This is already a data.dir structure. Doing nothing.

``` r

set.data.dir(CurrentProjectDirectory)
get.data.dir()
```

    ## [1] "D:/public/yang/MIND_STEP/New_test_fadnUtils"

### Required files

We request FADN data from DG-AGRI, which is delivered to us in csv
format. In order to work efficiently with R, we should convert the
csv-data to an r friendly format, this step is done with help of a
human-readable file, called `raw_str_map.file`. Both files are
necessary. `inst/examples` is the folder for use cases that contain
fadnUtils package examples and json files.

1.  FADN data in csv format: the data for loading
2.  A json file for extracting the variables

### Folder Structure

A working directory is specified arbitrarily by the user. This structure
helps data management and maintenance. The directory looks like this:

``` base

CurrentProjectDirectory/
+-- csv
+-- fadnUtils.metadata.json
+-- rds
\-- spool
    \-- readme.txt
```

- csv: CSV files are stored here
- fadnUtils.metadata.json: containing the mapping from the fadn.raw.rds
  to the fadn.str.rds data
- rds: placing r-data in the “rds” directory
- spool: keeping related files

## 2. Import CSV FADN data

First, we will import the data into an R-friendly format using the
fadnUtils package.

### Convert the csv data into raw r-data

The raw data will be added to a `rds` directory. We use a convenient
function from this package to convert the csv file into raw r-data.

``` r

fadn.data.dir <- "D:/public/data/fadn/lieferung_20210414/csv/"

# load data for country BEL and year 2009

convert.to.fadn.raw.rds(

      file.path =  paste0(fadn.data.dir, "BEL2009.csv"),

      sepS = ",",

      fadn.country = "BEL",

      fadn.year = 2009
      #keep.csv = T # copy csv file in csv.dir

      )

```

    ## File D:/public/data/fadn/lieferung_20210414/csv/BEL2009.csv does not exist. Exiting ...

    ## [1] FALSE

At any time, we can check for the current data dir, what csv files
(countries, year) are loaded.

``` r

show.data.dir.contents()
```

### Convert raw r-data into structured r-data

Then, We convert raw data into structured data. Broadly, there are 3
steps to including data in an R package:

1.  setting a structured data in the `structured` directory,
2.  checking the `raw_str_map.file` that all variables can be converted.

3.  converting the structured data successfully into `structured`
    directory.

#### Set a `structured` directory for saving the structured data

We set a `test` folder to placing the structured data.

``` r

rds.dir = paste0(get.data.dir(),"/rds/")
# set a structured name for for saving the structured r-data in rds.dir
new.str.name = "test"
# set a extraction_dir

dir.create(paste0(rds.dir, new.str.name), showWarnings = FALSE)

new.extraction.dir = paste0(rds.dir, new.str.name)
```

#### Check the variables in the `raw_str_map.file`

Before conversion it is recommended to use `check.column()` method,
ensuring that all variables in the`raw_str_map.file` can be converted.

``` r

list_vars = check.column(
              # a rds file or a csv file

              importfilepath = paste0(rds.dir, "fadn.raw.2009.BEL.rds"),

              # a json file
              jsonfile = "D:/public/yang/MIND_STEP/2014_after_copy.json",
              # write a new json file without unmatched variables
              rewrite_json = T,
              # save the new json in extraction_dir
              extraction_dir = new.extraction.dir)

```

#### Convert the raw data into structured r-data using the checked json file