warning("Either provide explicitly a fadnUtils.data.dir to the function orfirst to set the fadnUtils.data.dir using set.data.dir function. Exiting ....")
#' Updates selected elements of data stored in one DT with new one given in melted format
#'
#' The user provides the data.new: {id,variable,new value}. The function overwrites all existing id-column with the new values
#'
#' @param data.old The DT to update
#' @param data.new The data to insert. It must have three columns: {id,variable,new value}. E.g. data.new=data.table("id"=c(810001100105),"variable"=c("AASBIO_CV"),value=c(999999))
1. Allow the user to define csv configuration (delimiter, decimal point) and pass it to convert.to.fadn.raw.rds
4. Write a use case where the function convert.to.fadn.str.rds is used to recalculate the raw->str conversion (in order someone change the map manually)
6. Add the option of encrypting the rds files, see here https://stackoverflow.com/questions/52851725/how-to-protect-encrypt-r-objects-in-rdata-files-due-to-eu-gdpr
8. Provide the option to copy rds content from other data.dir directories
11. On 'manage_data_dir.R > overwrite.raw_str_map.file', re-run all convert.to.str.rds operations (currently only the replacement of the file is taking place)
12. Add the possibility for the user to add a column description of the fadn.raw data (providing a text file)
13. Add the following feature: An R-shiny application for browsing loaded fadn.raw. The user can start this with a simple command.
14. Throw a warning message if load.fadn.{raw,str} does not load anything. Say "No files found to belong to this country and years. Nothing loaded"
15. Create a filter.fadn.str. It will take a fadn.str and a filter(for data.table) expression and will keep only the records for info,costs, crops
16. On convert_data > convert.to.fadn.str.rds, use tryCatch() to report the error and not fail
17. In the raw_str_map.json file, provide the option to define factor levels for a variable
18. Provide the ability to delete country/years from the raw/str files
23. Give the possibility to load str.data passing some filtering for an ID field for str.fadn. In load.fadn.str.rds function
26. Save the SExxx variables to the dat.fadn list object (create an entry in raw_str_maps and add code in the convert.to.fadn.str.rds function)
27. Keep the raw.fadn.rds also in a long format (sparse matrix). Ability to select how to load (wide or long format). Need to know which variables are numeric and which are strings. Keep them in different DT. Long format will return a list with one DT with the numeric values and one with the string values.
CHANGES UNDER WAY:
21. Provide the ability to use an external raw_str_map file (use it and copy it to raw_str_maps).
22. Add the content of the raw_str_map used for convert.to.fadn.str.rds in the attribute of the rds data.
9. Provide full documentation of raw_str_map.json specification (already some in the doc of convert.to.fadn.str.rds function)
CHANGES COMPLETED: (In date-completed descending order / newer changes on the top)
28. Utility function: Update an fadn.raw.rds file with external data (rows of id-column-new value). Load the data and update them with the new values.
27. Give the possibility to load raw.data with row selection based on a criterion (examples: column_x == xxx; column_x>xxx, etc. ) In load.fadn.raw.rds function
24. Provide the ability when load.fadn.raw to pass a vector of columns to load (and discard the rest)
25. added a DEBUG mode for convert.to.fadn.str.rds (detailed information on what is calculated is shown)
20. Write a function that merges two raw_str_map.json files. It will be used if one wants to have a basic raw_str_map and wants to make marginal changes for a specific case (year or country)
19. Provide the ability to use more than one raw_str_map.json (create.data with a vector of raw_str_map.json files, show in contents the raw_str_map.json files, check data dir straucture changes, ,convert with specifying which)
2. Save loaded data to stored.rds.data.RData added store.rds.data function, restore.rds.stored.data function, also show the saved.data rds in the show.data.dir.contents)
3. Provide the option to provide a file with the description of the variables for the fadn.str.rds files (data.dir specific). Probably alter the raw_str_map.json specification
5. Create a folder spool, where the users can put relevant files
1. Make the map_definition an organic part of the fadnUtils.data.dir
2. Add the option of storing/not storing the original csv from DG AGRI in the data.dir folder
10. On 'load.fadn.str.rds', output the message "Loading from ..." with <cat> instead of <print>