IPTD-97
Slocum Glider Long-Term Database Solution
Description
MIKE: REWRITE
Request permission to use Rutgers University to host and analyze the data. (MIKE: Do we tie this in with Science & Research Wind Energy Project?)
Consider the use of an enterprise MySQL / PostGre type database solution. Increase the file size data upload limit, and continue testing.
-
Enterprise database is routinely backed up by DOIT.
-
Two staff members from WMS will have access to the database to upload data and work within the database.
-
Same two staff members will have database management privileges' to execute SQL code for updates, deletes, etc.
-
The software package R will be permitted to connect to the database to generate .fst files to run statistics on the data.
-
Having this data stored in an enterprise based database will minimize the reliance on a few individuals within the division.
-
MySQL / PostGre will hopefully be better suited for handling this large dataset for the long-term.
MILESTONES
Milestone 1: Determine if using MySQL is feasible once DOIT tweaks number of records / file size limit able to be uploaded. If not, is Postgres an option?
Milestone 2: Determine what user interfaces can be used to interact with the database that is being used. Ideally the user should be able to upload, download, and query data based on the front end interface tool that connects to the backend enterprise database.
Milestone 3: Have successful connections of 'R Software' to the database that can pull from the datasets and run statistical scenarios.
Project Justification
Typical data storage and analysis solutions such as MS Access and MS Excel are not an option due to the volume of records.
An attempt was made to use a DOIT MySQL database but the uploading of the data kept timing out. Yet to be seen if this can be tweaked to avoid that issue.
Currently using 2 local SQLite databases to store the data in 5 year increments. We consider this a temporary solution.