IPTD-88
Marine Fisheries Data Consolidation
Description
The Marine Fisheries Administration (MFA) conducts projects that gather biological and environmental data. Individual project leaders currently maintain their data in a variety of formats and often isolated from other staff who may need access to the data. The MFA wishes to consolidate its projects’ datasets by transitioning its data-management to a relational database system.
The MFA Data-consolidation Project is currently envisioned as being internally-managed and accomplished by MFA staff. Outside management options may be available that are not known to the MFA and are open to consideration. If internally managed, the project is limited by staff expertise and time. This is an anticipated long-term project with no deadline as there is no mandate above the MFA for completion, although the priority of the project within the MFA is moderate to high.
See NJOIT project documents at [TEAM] DEP Project Portfolio Managment [CHANNEL] Fish and Wildlife [FILE FOLDER] Marine Fisheries Data Consolidation
Project Justification
Consolidation will ensure that a data source is not neglected. Much of the data collected by MFA has been federally funded and should be available to anyone that asks. With the current structure of each project, whether currently collecting data or a project that only ran for a period of time, data is housed in a separate location and can easily be missed as a data source for information on a species or other environmental variables.
We got a warning message from Microsoft to update some IP addresses (see attached)
resolving the email warning
adjusting the contacts for future warnings (it only went to Kurt/Maria, but should include the program as primary leads)
Optimizing the costs, which appear to be steadily just above the predicted $2k per months, although the program has not achieved expected future use with more front-end updating, and with more queries and analysis, mostly via connection to Tableau.
UsageDate
CostUSD
Cost
Currency
2023-07-01
2360.561
2360.56071
USD
2023-08-01
2357.273
2357.273398
USD
2023-09-01
2281.222
2281.222274
USD
2023-10-01
2357.628
2357.628464
USD
2023-11-01
2308.768
2308.768486
USD
2023-12-01
2373.778
2373.777853
USD
2024-01-01
2369.491
2369.49108
USD
2024-02-01
2220.811
2220.810713
USD
2024-03-01
2378.07
2378.069851
USD
2024-04-01
2295.557
2295.556504
USD
2024-05-01
2367.413
2367.412523
USD
2024-06-01
2289.033
2289.032827
USD
27959.60468
Program meeting about weekly with newest MS contact Nic who is doing “over-the-shoulder” direction and teaching for Linda to feel capable with the no-caode means to modify, grow and maintain the solution.
follow-up meeting with Microsoft to discuss pain points for Greg and Linda to product delivered by Acuvate. Attending: Knute, Tony, Tish, Sebastian, Greg and Linda.
Tony laid out idea to provide some time from Sebastian who was given the videos and documentation from Acuvate.
Linda expressed that the solution is overly complicated with merging of tables that she can’t figure out how to make work. Tying in all Species makes sense, other table merges do not (Tow with Lengths and Haul).
Sebastian is able to dedicate about an hour per week for a few weeks with offline sharing to get somewhere new. What is the outcome/deliverable we are seeking? Greg: Script-less management of the dataflow that DEP can manage and a re-worked mapping of the data (possibly scrap the model already built).
Sebastian asked Tony about what training or existing NJ support is possible for Greg and Linda. The answer is a little unclear, but Tony suggested seeking ongoing support under the new EA.
Sebastian to meet with Linda (recorded and others optional) the week of Monday May 8th. First meeting for scoping and to generate a plan. Tony to set meeting.
Aprill 4- final acceptance sign-off of Acuvate product with SHI.
Meeting about a week ago with Microsoft (Tony Pagano, Tyler McLaughlin and Tish Scarborough) to go over the expectations for no-code development of pipelines. We found the product was delivered with a good bit of custom code that we are not confident in replicating. Tony will follow-up with Sebastian who ran the no-code POC and promised a follow-up and ongoing support to ensure our confidence with the tools.
Tony to meet with Sebastian Friday and bring a plan for progress back to our DEP team and Marine Fisheries.
meeting 10/27- General agreement to proceed. Recognized as a learn-as-we-go effort, new to DOIT, and NJOIT. Ginger inquired about how this might help many others and might be worth more general DEP funding. Ginger and Pete talked through cost, with Pete backing up a first year if the program had trouble funding it (estimated at $2k per year).
Knute will pursue the mini-bid/ethics clearance and LSAR and call people together for the next useful step toward a DPI30 effort.
Questions for Tony at the next Customer Success session:
(Or for Kayran) How much does the existing support contract apply to needs of a Fisheries team with help building out an Azure/Synapse world (where a partner may not complete all in DPI30)?
How do we secure such support if it is not covered or if we exceed already paid support?
Currently awaiting costs from Microsoft.
Azure Synapse Training - day 1 https://microsoft-my.sharepoint.com/:v:/p/kabasali/EdWyPFk5QvVJgobqm106HRQBKN0ywoRy_dcU_SnR6PHPxA
day 2 - https://microsoft-my.sharepoint.com/:v:/p/v-tzawar/Eac8H9HhOsdGmqu_eX4YMp4B4LOd7Y-p2usEr_j0JqYdgA
Worth some conversation about potential to partner with other states and orgs including at these links:
Data - Atlantic States Marine Fisheries Commission (asmfc.org)
NEAMAP (see log of state contacts halfway down here List of potential survey participants to NEAMAP surveys
published program info March 2022 NJDEP Division of Fish & Wildlife - Studying the Delaware River - 2019 Report