\title{Converts an fadn.raw.rds file to fadn.str.rds file using a raw_str_map.json file}
\usage{
convert.to.fadn.str.rds(
fadn.country = NA,
fadn.year = NA,
raw_str_map.file = NULL,
force_external_raw_str_map = FALSE,
str.name = NULL,
DEBUG = F
)
}
\arguments{
\item{fadn.country}{string with the country to extract the str data}
\item{fadn.year}{the year to extract the structured data}
\item{raw_str_map.file}{the full path to the raw_str_map file.}
\item{DEBUG}{if TRUE, prints more details on the conversion process}
\item{str.short_name}{the short name of the str data. No spaces and text up to 20 characters}
}
\value{
Saves the rds.str.fadn and returns TRUE if everything goes well
}
\description{
The raw_str_map.json specification is as follows:
}
\details{
{
"id": { "COLUMN in every list member in RDS": "COLUMN IN CSV", ....},
"info": { "COLUMN in info RDS": "COLUMN IN CSV", ....},
"livestock": {}
"crops": {
"CROP NAME 1": {"description": "description of crop name", "columns": {"VARIABLE NAME": COLUMN IN CSV", ....} },
"CROP NAME 2": {"description": "description of crop name", "columns": {"VARIABLE NAME": COLUMN IN CSV", ....} },
....
}
}
The structure of the str.dir:
- A data.dir can hold more than one extractions.
- Each extraction has a short name (20 or less characters, whitespace is not allowed)
- Each extraction is stored in the data.dir/rds/<extraction_name>
- That folder contains the following files:
+ raw_str_map.json: the raw_str_map
+ fadn.str.<4-digit YEAR>.<3-letter COUNTRY>.rds: the extracted data
Notes:
1) The computed RDS file contains a list structure with the following keys: info, costs, livestock-animals and crops
All are data.tables. For all of them, the first columns are those that are contained in the "id" object
"info" and "costs" are in table format, i.e. each farm is one row and data is on columns, as defined in the
related raw_str_map.json file.
"crops" and "livestock-animals" are in wide data format (https://tidyr.tidyverse.org/), where one farm lies accross many rows, and each
row is a farm-crop-variableName-value combination
2) In $id, $info and $costs, "COLUMN IN CSV" can have two forms
i) a single column name in the fadn.raw csv file or a combination, e.g. "K120SA+K120FC+K120FU+K120CV-K120BV"
ii) the form of an object {"source": "the column in the csv", "description": "a description of what this column is about"}
3) We attach certain attributes that are useful for identifying informations:
i) In $info and $costs, the attribute "column description" provide information of the formula and the description of each column
ii) In $crops and $livestock-animals, the attribute "$crops.descriptions" and "$livestock.descriptions", provide the description of each CROP contained there
iii) In $crops and $ the attribute "$column.formulas" provide the formulas used in order to derive the VALUE
% Please edit documentation in R/check_column_names.R
\name{nested_var}
\alias{nested_var}
\title{Check a objest in the json file}
\usage{
nested_var(var, rds)
}
\arguments{
\item{var}{A object or category of raw json.}
\item{rds}{All variables/column names in fadn.raw.rds file.}
}
\value{
A list of multiple objects. This list combines no machted variables and the modified json for the chosen object/category.
}
\description{
This function checks the node of chosen object/category for the json file and find out the variables
which are in json file but not in fadn.raw data file.
Returning two lists: unmatched variables/column names and modified json.
If unmatched variable exists, this variable will be deleted from the json list.
}
\details{
A json file has 6 parent objects/categories: "id", "info", "costs", "crops", "subsides", "livstock". This function checks all objects inside the parent object.
\title{Merges two raw_str_map files and returns either a list or a file}
\usage{
raw_str_map.merge(
source.raw_str_map.file = NULL,
new.raw_str_map.file = NULL,
return.file = F
)
}
\arguments{
\item{source.raw_str_map.file}{the filename of the source raw_str_map. It must be relative the raw_str_maps of the current data.dir}
\item{new.raw_str_map.file}{the filename of the mask raw_str_map. It will replace any entries of the source file. It must be relative the raw_str_maps of the current data.dir}
\item{return.file}{If set to T, a temporary full file path that contains the merge is returned. Otherwise a list with the contents of the merge is returned}
}
\value{
FALSE in case of problem / if return.file=T, the temporary full path of a file that contains the merged result in json / A list with the contents of the merge if return.file=F
}
\description{
All entries in the new.raw_str_map file replace those on the source.raw_str_map file
}
\details{
Both files must be relative to the current data.dir/raw_str_maps
\title{Updates selected elements of data stored in one DT with new one given in melted format}
\usage{
update_elements.DT(data.old, data.new)
}
\arguments{
\item{data.old}{The DT to update}
\item{data.new}{The data to insert. It must have three columns: {id,variable,new value}. E.g. data.new=data.table("id"=c(810001100105),"variable"=c("AASBIO_CV"),value=c(999999))}
}
\value{
a DT with the updated values
}
\description{
The user provides the data.new: {id,variable,new value}. The function overwrites all existing id-column with the new values
1. Allow the user to define csv configuration (delimiter, decimal point) and pass it to convert.to.fadn.raw.rds
4. Write a use case where the function convert.to.fadn.str.rds is used to recalculate the raw->str conversion (in order someone change the map manually)
6. Add the option of encrypting the rds files, see here https://stackoverflow.com/questions/52851725/how-to-protect-encrypt-r-objects-in-rdata-files-due-to-eu-gdpr
8. Provide the option to copy rds content from other data.dir directories
11. On 'manage_data_dir.R > overwrite.raw_str_map.file', re-run all convert.to.str.rds operations (currently only the replacement of the file is taking place)
12. Add the possibility for the user to add a column description of the fadn.raw data (providing a text file)
13. Add the following feature: An R-shiny application for browsing loaded fadn.raw. The user can start this with a simple command.
14. Throw a warning message if load.fadn.{raw,str} does not load anything. Say "No files found to belong to this country and years. Nothing loaded"
15. Create a filter.fadn.str. It will take a fadn.str and a filter(for data.table) expression and will keep only the records for info,costs, crops
16. On convert_data > convert.to.fadn.str.rds, use tryCatch() to report the error and not fail
17. In the raw_str_map.json file, provide the option to define factor levels for a variable
18. Provide the ability to delete country/years from the raw/str files
23. Give the possibility to load str.data passing some filtering for an ID field for str.fadn. In load.fadn.str.rds function
26. Save the SExxx variables to the dat.fadn list object (create an entry in raw_str_maps and add code in the convert.to.fadn.str.rds function)
27. Keep the raw.fadn.rds also in a long format (sparse matrix). Ability to select how to load (wide or long format). Need to know which variables are numeric and which are strings. Keep them in different DT. Long format will return a list with one DT with the numeric values and one with the string values.
CHANGES UNDER WAY:
21. Provide the ability to use an external raw_str_map file (use it and copy it to raw_str_maps).
22. Add the content of the raw_str_map used for convert.to.fadn.str.rds in the attribute of the rds data.
9. Provide full documentation of raw_str_map.json specification (already some in the doc of convert.to.fadn.str.rds function)
CHANGES COMPLETED: (In date-completed descending order / newer changes on the top)
28. Utility function: Update an fadn.raw.rds file with external data (rows of id-column-new value). Load the data and update them with the new values.
27. Give the possibility to load raw.data with row selection based on a criterion (examples: column_x == xxx; column_x>xxx, etc. ) In load.fadn.raw.rds function
24. Provide the ability when load.fadn.raw to pass a vector of columns to load (and discard the rest)
25. added a DEBUG mode for convert.to.fadn.str.rds (detailed information on what is calculated is shown)
20. Write a function that merges two raw_str_map.json files. It will be used if one wants to have a basic raw_str_map and wants to make marginal changes for a specific case (year or country)
19. Provide the ability to use more than one raw_str_map.json (create.data with a vector of raw_str_map.json files, show in contents the raw_str_map.json files, check data dir straucture changes, ,convert with specifying which)
2. Save loaded data to stored.rds.data.RData added store.rds.data function, restore.rds.stored.data function, also show the saved.data rds in the show.data.dir.contents)
3. Provide the option to provide a file with the description of the variables for the fadn.str.rds files (data.dir specific). Probably alter the raw_str_map.json specification
5. Create a folder spool, where the users can put relevant files
1. Make the map_definition an organic part of the fadnUtils.data.dir
2. Add the option of storing/not storing the original csv from DG AGRI in the data.dir folder
10. On 'load.fadn.str.rds', output the message "Loading from ..." with <cat> instead of <print>