Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r setup, include=FALSE}
getwd()
knitr::opts_knit$set(root.dir = 'D:/data/fadn/lieferung_20210414/yang/FadntoCapri')
```
# FadntoCapri
<!-- badges: start -->
<!-- badges: end -->
The goal of FadntoCapri is to convert FADN data into CAPRI. This project bases on the `fadnUtils` package.
1. With the help of internel or externel JSON files, `fadnUtils` package can be used to extract `crops` data from raw FADN data .
1. Thanks to Dimitrios Kremmydas's R script, the `animals` data can be extracted from raw FADN data.
1. Convert the `crops` and `animals` data into different NUTS version in both directions.
1. Export GDX files for CAPRI.
## Installation
To get started you need the following softwares installed on your computer: `git`, `R` and `Rstudio`.
You can open the git bash prompt, then clone this repository using the following command line:
```bash
git clone https://git-dmz.thuenen.de/mindstep/fadntocapri.git
```
It is also possible to create a new Rstudio project with git version control. In Rstudio Select File -> New Project -> Version Control -> Git, then provide the repository HTTPS link: https://git-dmz.thuenen.de/mindstep/fadntocapri.git, select the R workspace folder and create the project. RStudio now copies (clone in Git terms) the content of the repository to your project folder. The content of the Gitlab repository should now appear in the Files pane of RStudio.
## Usage
### Setup and load Libraries
* Loading libraries
Make sure you have `fadnutils` installed.
Using `devtools` to install `fadnutils`. If you did not install this package, please just run the following command at the root of your project:
```{r, results='hide', message=FALSE}
# Install fadnutils from thuenen internal gitlab
devtools::install_git("https://git-dmz.thuenen.de/mindstep/fadnutilspackages",
credentials = git2r::cred_user_pass("yang", "Thu_Y2021"))
# load libraries
requiredPackages = c('fadnUtils','data.table', 'devtools','jsonlite', 'ggplot2',
'gdxdt', 'tidyverse', 'xlsx', 'gdxrrw','Hmisc')
for(p in requiredPackages){
if(!require(p,character.only = TRUE)) install.packages(p)
library(p,character.only = TRUE)
}
```
* Setting GAMS path
It is prepare for exporting GDX files.
```{r}
igdx("d:/gams/win64/34.3")
```
* Setting FADN project directory and list of Europe country codes
You can set the path for `fadnUtils` package if you have not set, then you can create a new path using `create.data.dir()` command. The European Union has 28 member countries, CAPRI project focuses on EU countries, so we have to set all of the country codes for Europe, including 3-letter codes as below.
```{r, results='hide',message=FALSE}
# Set FADN project directory ----
CurrentProjectDirectory = "D:/data/fadn/lieferung_20210414/yang/fadn_work_space"
# Ceate a data.dir
create.data.dir(folder.path = CurrentProjectDirectory)
# Once the data.dir is created, we must declare that we are working with it
set.data.dir(CurrentProjectDirectory)
rds.dir = paste0(get.data.dir(),"/rds/")
# Set EU list
EU_list <- c("AUT", "BEL", "BGR", "HRV", "CYP", "CZE", "DNK",
"EST", "FIN", "FRA", "DEU", "GRC", "HUN", "IRL",
"ITA", "LVA", "LTU", "LUX", "MLT", "NLD", "POL",
"PRT", "ROU", "SVK", "SVN", "ESP", "SWE")
```
* Loading functions from R script `myfun_fadn.R`
This script is responsible for grouping by different variables (NUTS versions, organic or non-orgnic, etc.), converting the `crops` data and getting `animals` data.
```{r}
# Load functions
source("myfun_fadn.R")
```
### `Crops` data for CAPRI
Using `output_GDX_crops` function loads raw data, extracts crops data, computes orgnic and non-org, aggregtes different variables and exports `crops` FADN data.
```{r}
output_GDX_crops <- function (fadn.countries ){
# Convert and load FADN data, save the str data in path: D:\data\fadn\lieferung_20210414\yang\fadn_work_space\rds\crops
cat("Convert and load FADN str data for:", fadn.countries)
fadn.str.crops <- convert.load.str.crops(countries = fadn.countries)
cat("Convert FADN str data to NUTS Version 2016 for country:", fadn.countries)
# Export crops gdx file
# step 1: Convert str data to NUTS 2016
fadn.str.crops <- fadnUtils::NUTS.convert.all(fadn.str.crops,fadn.countries, 2016)
# filter LEVL
fadn.str.crops.levl <- fadn.str.crops %>%
mutate(ORGANIC=case_when(
ORGANIC=="org-2" ~ "Organic",
TRUE ~ "Conventional"))%>%
mutate(value2 = WEIGHT*VALUE) %>%
filter(VARIABLE == "LEVL" ) %>%
select(-(TF8:SIZ6),-WEIGHT, -ALTITUDE)
# step 2: export a gdx
# export DEU gdx took 1.25 mins
group_by_lst <- c("COUNTRY","REGION","NUTS1","NUTS2","NUTS1_final", "NUTS2_final", "EU")
crops.groupby <- lapply(seq_along(group_by_lst),
function(i) fadn.filter(fadn.str.crops.levl,
group_by_lst[i],
"CROP")) %>%
bind_rows() %>%
mutate(REG_TYPE = case_when(REG_TYPE == "COUNTRY" ~"MS",
REG_TYPE == "REGION" ~ "FADN_REGION",
REG_TYPE == "NUTS1" ~ "NUTS1_ORG",
REG_TYPE =="NUTS2"~ "NUTS2_ORG",
REG_TYPE =="NUTS1_final"~ "Xinxin_NUTS1",
REG_TYPE =="NUTS2_final" ~"Xinxin_NUTS2",
TRUE ~ REG_TYPE)) %>%
mutate_if(is.factor, as.character)
col_names_crops <- colnames(crops.groupby)
# library(gdxrrw)
# igdx("d:/gams/win64/34.3")
cat("Export the gdx: ", getwd(), "/gdx/crops_LEVL_new.gdx",sep = "")
# write gdx: levl: value*WEIGHT
writegdx(dt = crops.groupby,
gdx = paste0(getwd(), "/gdx/crops_LEVL_new.gdx"),
name = "DataOut",
valcol= "sum_Value",
uelcols= col_names_crops[!col_names_crops %in% "sum_Value"],
type="parameter")
}
```
Call funtion to export GDX files for crops of EU country.
```{r}
output_GDX_crops(fadn.countries = "all")
```
### `Animals` data for CAPRI
All the information about `animals` comes from table J (collection of tables are provided by FADN). `output_GDX_animals` includes loading raw animals data from FADN data, cleaning raw data, and exporting final animals data as GDX format.
```{r}
output_GDX_animals <- function( convert.str.animal = FALSE, fadn.countries = "all" ) {
if (convert.str.animal){
# converting animal str data took 815.39s
system.time(source("animals.R"))
fadn.str.animals.df <- animal.dt
} else{
# loading the animals data took 316.89s
fadn.str.animals.df <- readRDS(paste0(get.data.dir(),"/rds/str_dir/fadn.str.animal.rds"))
}
# Convert str data to NUTS 2016 ---
fadn.str.animals <- fadnUtils::NUTS.convert.all(fadn.str.animals.df,fadn.countries,2016)
fadn.str.animal.an <- fadn.str.animals %>%
mutate(ORGANIC=case_when(
ORGANIC=="org-2" ~ "Organic",
TRUE ~ "Conventional"
),YEAR=as.factor(YEAR)) %>%
mutate(value2 = WEIGHT*value) %>%
filter(variable=="AN") %>%
rename(VARIABLE= variable) %>%
select(-(TF8:SIZ6),-WEIGHT)
group_by_lst <- c("COUNTRY","REGION","NUTS1","NUTS2","NUTS1_final", "NUTS2_final", "EU")
animals.groupby <- lapply(seq_along(group_by_lst),
function(i) fadn.filter(fadn.str.animal.an,
group_by_lst[i],
"ANIM")) %>%
bind_rows() %>%
mutate(REG_TYPE = case_when(REG_TYPE == "COUNTRY" ~"MS",
REG_TYPE == "REGION" ~ "FADN_REGION",
REG_TYPE == "NUTS1" ~ "NUTS1_ORG",
REG_TYPE =="NUTS2"~ "NUTS2_ORG",
REG_TYPE =="NUTS1_final"~ "Xinxin_NUTS1",
REG_TYPE =="NUTS2_final" ~"Xinxin_NUTS2",
TRUE ~ REG_TYPE)) %>%
mutate_if(is.factor, as.character)
col_names_animals <- colnames(animals.groupby)
# library(gdxrrw)
# igdx("d:/gams/win64/34.3")
cat("Export gdx: ",paste0(getwd(), "/gdx/animals_LEVL_converted.gdx") )
# write gdx: levl: value*WEIGHT
writegdx(dt = animals.groupby,
gdx = paste0(getwd(), "/gdx/animals_LEVL_converted.gdx"),
name = "DataOut",
valcol= "sum_Value",
uelcols= col_names_animals[!col_names_animals %in% "sum_Value"],
type="parameter")
}
```
Call funtion to export GDX files for animals of Germany.
```{r}
output_GDX_animals(convert.str.animal = FALSE, fadn.countries = "DEU")
```