integrate.r

##
##    Programme:  Oceans Accounts_Version2.r
##
##    Objective:  
##                From: Neville Smith <nevilles@spc.int> 
##                Sent: Friday, 26 September 2025 2:36 pm
##                To: James Hogan <jamesh@spc.int>
##                Subject: RE: REsponse requested: Ocean Accounting for Integrated Ocean Management and NOP implementation
##                
##                Hi James,
##                
##                Generally positive indeed. I would suggest replacing Fiji and/or adding a fourth with New Caledonia, as it gives us territory/francophone on top of all of the other benefits/pros identified for Fiji.
##                
##                Happy to discuss further.
##                
##                Cheers,
##                Neville 
##                From: James Hogan <jamesh@spc.int> 
##                Sent: Friday, 26 September 2025 2:47 AM
##                To: Neville Smith <nevilles@spc.int>
##                Subject: Fw: REsponse requested: Ocean Accounting for Integrated Ocean Management and NOP implementation
##                
##                Some really good feedback from Sally!
##                
##                Bat regards, 
##                James
##                
##                –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––- 
##                                   
##                
##                James Hogan
##                Senior Marine Resource Economist
##                Économiste principal des ressources marines
##                Pacific Community | Communauté du Pacifique
##                CPS – B.P. D5 | 98848 Noumea, New Caledonia | Nouméa, Nouvelle-Calédonie
##                Tel: (+64) 275 997 999
##                E: jamesh@spc.int | Website | Twitter | LinkedIn | Facebook | YouTube | Instagram
##                
##                                   
##                –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––-
##                As part of our emissions reduction strategy, please only print this email if necessary
##                Dans le cadre de notre stratégie de réduction des émissions, merci d'imprimer cet e-mail uniquement si nécessaire
##                                   
##                
##                ________________________________________
##                From: Sally Bailey <sallyb@spc.int>
##                Sent: Friday, September 26, 2025 12:08:44 AM
##                To: James Hogan <jamesh@spc.int>; Jens Kruger <jensk@spc.int>; Tony Kabasi <tonyk@spc.int>; Lisa Green <lisag@spc.int>; Sachindra Singh <sachindras@spc.int>; Litia Gaunavou <litiag@spc.int>
##                Cc: Duncan Hume <duncanh@spc.int>
##                Subject: REsponse requested: Ocean Accounting for Integrated Ocean Management and NOP implementation 
##                
##                Bula James (and colleagues)
##                
##                Thanks again for the paper and the discussion last Monday. Once again apologies for the delayed response, as you know I came straight back into the EQAP meeting and then 
##                into organising PIOC, EQAP endorsement, and Adaptation Futures, so I haven’t been able to give the paper the time it deserves. That said, I really appreciate you putting 
##                something together that we can all discuss, hence copying in Tony, Jens, Lisa (who I briefly caught up with in Noumea last week), and Sachin.
##                
##                My initial thoughts as discussed,  is that this work should be reframed from a regional initiative into a country codesign process, where Ocean and Environmental Accounts 
##                are embedded as decision support tools (DSTs) for both IOM and MSP. Fiji, Palau, and Cook Islands could provide the ideal test cases, given their existing governance 
##                frameworks and political interest. By grounding OA in national priorities and planning systems, SPC can ensure these accounts deliver real policy impact, strengthen 
##                access to blended finance opportunities, and build durable national ownership.
##                
##                Rather than a region wide statistical focus, the opportunity is to codesign Ocean and Environmental Accounts as DSTs with at least one country per sub-region. This would 
##                allow us to embed them directly into MSP and IOM processes and test their utility in diverse contexts. 
##                
##                Suggested pilots:
##                •	Fiji (Melanesia): EU-funded for IOM, recently completed the MSP Global Rapid Assessment, and working with GOAP on OA. The Permanent Secretary has indicated  interest, 
##                   which makes Fiji a natural pilot for testing OA integration into both IOM frameworks and MSP planning.
##                
##                •	Palau (North Pacific): MSP funded by US and MFAT. While there have been engagement challenges, OA could be positioned as the evidence base for evaluating trade-offs in 
##                   MSP zoning and for demonstrating the economic value of conservation.
##                
##                •	Cook Islands (Polynesia): MFAT funding MSP, EU funding IOM, and the Marae Moana Act provides a legal framework for integrated management. This makes Cooks a ready 
##                   platform to embed OA as a cross-sector DST that strengthens both MSP zoning and IOM policy monitoring. I know Maria has reached out to you, which shows interest, 
##                   reinforcing momentum.
##                
##                A three-country approach would allow SPC to test and refine methodology in different governance and ecological contexts before scaling up. We are under time pressure, given 
##                the US DoS project ends September 2026, so plan B would just to do one and I suggest that is Fiji as it is easily accessible and if something not moving easy to go for a 
##                coffee or Talanoa over bowl of kava.
##                
##                •	For MSP: OA can be converted into spatially explicit data layers to inform zoning, sector negotiations, and trade-offs (tourism, fisheries, conservation, etc.).
##                •	For IOM: OA can provide the evidence base for National Ocean Policies (NOPs), track how economic sectors impact ecosystems, and measure progress toward IOM outcomes.
##                
##                By focusing on three contexts, we can develop differentiated approaches:
##                
##                •	In Fiji, OA could complement IOM governance frameworks while supporting emerging MSP.
##                •	In Palau, OA could strengthen MSP by quantifying ecosystem service values and supporting the emerging NOP.
##                •	In Cook Islands, OA could sit at the interface of MSP (zoning) and IOM (Marae Moana governance), linking both to sustainable finance opportunities.
##                
##                The trick is to position accounts as practical policy tools, not just technical outputs. Data generated could support:
##                
##                •	Accessing finance (climate, blue carbon, biodiversity credits).
##                •	Demonstrating ecosystem value in economic terms to strengthen the case for conservation and sustainable use.
##                •	Reporting progress toward Blue Pacific Continent, SDG14, Paris Agreement, 30x30, and other commitments.
##                
##                SPC’s role should be clearly framed as facilitator and technical enabler. The donors I work with are clear they want national governments in the driving seat, with SPC 
##                providing technical and coordination support. Therefore, I beleive OA should be designed to strengthen national IOM and MSP institutions, with codesign processes engaging 
##                statistical offices, environment ministries, ocean governance bodies, and communities to ensure both legitimacy and usability.
##                
##                Although this work should be country focused, there is a strong opportunity to create a regional learning loop. By linking Fiji, Palau, and Cook Islands through a structured 
##                Community of Practice on Ocean Accounting for MSP/IOM, or through the existing IOM SC which includes countries beyond the EU project, SPC can ensure that lessons, methods, 
##                and tools are shared across contexts. 
##                
##                This will:
##                •	Help identify what is transferable between different governance systems.
##                •	Allow scaling and replication in other Pacific countries.
##                •	Build a regional narrative on how OA supports the Pacific’s collective priorities (Blue Pacific strategy, SDG14, 30x30, climate resilience, and finance mobilisation).
##                
##                This also fits with donor requirements in building capacity and regional coordination and collaboration in both the CR-MSP funded by US DoS and CABSMART – MSP funded by 
##                NZMFAT
##                
##                I would like to move this forward with an internal discussion. I know availability is a challenge: you are on leave, Lisa is in the UK, Sachin won’t be at PIOC, and Jens 
##                and Tony are busy with other meetings. However, it would be good if @Jens Kruger @Sachindra Singh @James Hogan@Lisa Green@Litia Gaunavou @Tony Kabasi you could all respond 
##                with your thoughts on James’ paper and my suggestions above.
##                
##                Hopefully, we can use the margins of PIOC to meet with GOAP, Fiji and Cook Islands representatives, and agree next steps. From there, we can continue momentum as a small 
##                working group, also linking with the new IOM Adviser once they start, and with the upcoming consultancy Jens has circulated on maritime work in Solomon Islands.  
##                
##                In addition, I have copied in Duncan who has just started as Data Management Adviser and he will be at PIOC.
##                
##                Vinaka
##                Sally
##                Sally Bailey
##                Marine Spatial Planning Coordinator 
##                Coordinatrice - Planification spatiale marine
##                Pacific Community
##                SPC - Private Mail Bag - Suva, Fiji
##                Level 2 - Lotus Building, Ratu Mara Road, Nabua. (18°07'16.4"S 178°27'32.2"E)
##                Tel: (679) 3381377 Ext 35463 | Direct line: +679 3379463
##                Email: sallyb@spc.int | Website | Twitter | LinkedIn | Facebook | YouTube | Instagram 
##                
##                
##    Hot Tips:   (1)   I trained the regression dataset on the full raster including the sea - I got a lot of observations of the sea.
##    for next:         Next time, subset the rasters on the Pacific Coastline dataset from the Pacific Data Hub so that all the samples
##    time    :         are of land.
##                
##                (2)   Spend a lot more effort cutting out the land component of the data. The next iteration of this programme is going 
##                      to bring all of the Pacific land together and sample from that, rather than just train on a subset of New Caledonia
##                      and mostly sampling the ocean.
##                
##                
##                Dropping Fiji for the moment because having difficult getting its satelite data from STAC - issues with 180degrees cutting the country
##                
##                
##                So, we're going to do all this again for four countries: Fiji, Palau, Cook Islands and New Caledonia
##                
##                Alrighty... This code is being saved in a separate branch "Version2MoreCountries"
##                
##                Lets get into it....
##                
##                
##                ON DIGITAL EARTH PACIFIC
##                DEP data is building and developing, but still has aspects of under-developed documentation about it. For example, all of its datasources are here:
##                https://data.digitalearthpacific.org/ but there is no documentation for what are each. 
##
##                The closest that comes to documentation is: https://digitalearthpacific.org/#/applications
##
##
##                ON THE DIFFERENCE BETWEEN MATRICIES AND RASTERS
##                https://geobgu.xyz/r/matrices-and-rasters.html#rasters
##
##                
##
##    Author:     James Hogan, Senior Marine Resource Economist, 8 October 2025
##
##
   ##
   ##    Clear the memory
   ##
      rm(list=ls(all=TRUE))
      options(max.print=999999)
      options(scipen = 100)

   ##
   ##    Core libraries
   ##
      library(ggplot2)
      library(plyr)
      library(stringr)
      library(reshape2)
      library(lubridate)
      library(calibrate)
      library(Hmisc)
      library(RColorBrewer)
      library(stringi)
      library(sqldf)
      library(extrafont)
      library(scales)
      library(RDCOMClient)
      library(extrafont)
      library(tictoc)   
      
      library(sysfonts)
      library(showtext)

   ##
   ##    Special Libraries
   ##      
      library(tensorflow)
      library(GPUmatrix)
      library(Matrix)
      library(keras3)
      library(terra)
      library(raster)
      library(sf)
      library(parallel)   
      library(stats)   
      library(MASS)
      library(bigmemory)
      library(dym)         # This library is derived from here: https://osf.io/hgfjq/files/5kqyf
      
   ##
   ##    Set working directory
   ##
      setwd("c:\\Git_Projects\\Ocean_Accounts")
      setwd("C:\\From BigDisk\\GIT\\Ocean_Accounts")

      ##
      ## ESA data cames from here: https://maps.elie.ucl.ac.be/CCI/viewer/download.php  and is their ESACCI-LC-L4-LCCS-Map-300m-P1Y-2015-v2.0.7.tif file which is their world land use for 2015
      ##

   ##
   ##    Run "Programmes/DEP_Play_Code_Version3.py" (modified for extracting island level data) in Python.
   ##    It will download some code shape files into the Data_Spatial directory
   ##
      ##
      ##    Estimate the bounded boxes for Fiji, Palau, Cook Islands and New Caledonia EEZ so these can be passed to "Programmes/DEP_Play_Code_Version3.py" for data extraction
      ##       Dropping Fiji for the moment
      ##
         source("Programmes/Estimate_bounded_boxes_Version4.r") # I moved onto V2 for issues associated with splitting countries into islands. Read programme notes.
                                                                # So... Version 2 chopped the countries into islands and brought each down individually. That worked, in the sense that data was 
                                                                # able to be downloaded, but it failed because it didn't get the full complexity of the islands in the likes of Fiji.
                                                                # Version 3 kept the countries, but wasn't able to download data from New Caledonia becuase it was too big.
                                                                # Vertion 4 will revert back to Version2, and break the countries up into islands, and scope 10km around each to get the complexity
                                                                #    hopefully coming through the ESA land use measures
         
      ##
      ##    Thats all the data pulled down. Now I've got to make an index which relates the 10x10 spatial location with the 300x300 land use classification for each of the islands of the 
      ##       countries.
      ##
      ##    So, I need to:
      ##       (1) Chop out the sentinel-2 (S2) data into the shape of the islands, together with a 1 km buffer around the land mass. This excess RASTER data is not needed and is file bloat.
      ##       (2) Chop out the ESA 2 data for shape of the islands - simplify, simplify, simplify.
      ##       (3) Figure out the ESA value associated with each S2 cell so those S2 cells inherit the ESA land value
      ##       (4) Draw a sample of GRB S2 values and their ESA values from the ENTIRE dataset for all four countries. 
      ##
      ##           Two concepts spring to mind:
      ##             (a) The size of the land use varies across the region. To get a good land use estimate, I should stratify by land use and draw a sample proportional to the land use totals so 
      ##                 each land use has a proportional liklihood of being in the regression data set.
      ##
      ##             (b) Second, each country also varies significantly in collect land size. If I draw a sample, even if proportional to Land Use, it will be biased towards the RGB colours from large countries
      ##                 which will dominate the sample.
      ##
      ##           So, I need to draw a stratified random sample from country S2 values with BOTH land use and Country are the strata.
      ##

         ##
         ## (1) Chop out the sentinel-2 (S2) data into the shape of the islands 
         ##
            source("Programmes/Sentinel_2_Chop.r")


         
   
#      source("Programmes/ESA and Sentinel Data_Version3.r") # This was the prototype programme for the regression analysis

#      source("Programmes/Draw_Samples.r")         # This draws two mutually exclusive random samples of the Sentinel-2 data for regression and testing
      source("Programmes/Draw_Samples_Version2.r") # This draws two mutually exclusive random samples of the Sentinel-2 data for regression and testing, but now uses the Pacific Coastline data to exclude the sea
      
      source("Programmes/Create_Test_Set.r")       # This parallel processes a sample of sentinel-2 data against the ESA data to identify potential ESA land use for testing the model.
      source("Programmes/Create_Regression_Set.r") # This parallel processes a sample of sentinel-2 data against the ESA data to identify potential ESA land use for regression.
   
      source("Programmes/Logistic_Regression.r")           # This estimates the Sentinel-2 RGB values associated with ESA land use. It estimates using the regression set, and tests on the test data.
      source("Programmes/Logistic_Regression_to_Test.r")   # This applies the Logistic_Regression models to the test data. I separated this from the above programme because I split the regression
                                                           # and test dataset creation across machines. So the machine that makes the regression data can carry on and estimate the models while the 
                                                           # machine that makes the test data can finish and then run this programme.
      source("Programmes/Explore_Logistic_Results.r")      # Creates a picture of land use using the test data

      source("Programmes/Create_a_TimeSeries.r")           # Apply the logistic regression models to timeseries of land to estimate area.