-
Notifications
You must be signed in to change notification settings - Fork 4
/
01_interpolate_forecast_det.Rmd
131 lines (112 loc) · 7.27 KB
/
01_interpolate_forecast_det.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
# Interpolating model outputs
```{r setup01, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
```
## Introduction
The "internal" file format for harp is SQLite. SQLite allows fast access to the data and the ability to filter exactly what you want to read. Once our data are in SQLite format, we don't have to convert them again.
In most cases you also need to interpolate forecast data to station locations, and harp enables you to read the data, interpolate it, and write the interpolated data in one function. This can be done for grib files, FA files, netcdf files (from MET Norway) and vfld files (which are already interpolated). All of the file IO in harp is handled by the package harpIO.
## File templates
NWP files typically have information from any of the date, lead time, parameter, member number (for ensemble forecasts) and potentially more in the file name and / or directory structure. This means that the exact file names are never the same though they will always have the same structure. harp provides functionality for using templates for file names, and also has a number of built in templates for known formats. You can list these using `show_file_templates()` and see the way that the templates are built.
```{r}
library(harp)
library(here)
show_file_templates()
```
Since the output is typically truncated you can see one of the templates in full by supplying the row number in the original output. For example, to see the template for vfld files
```{r}
show_file_templates(23)
```
## Deterministic forecast data
Let's go ahead and read some data. In the data directory you will find a vfld directory and there you will find a directory for AROME_Arctic_prod. Under that directory you will find vfld files for one day of forecasts. To read the date we will use the function `read_forecast`. Each of the arguments are annotated so that you understand what they are for.
```{r message=FALSE}
read_forecast(
start_date = 2019021700, # the first forecast for which we have data
end_date = 2019021718, # the last forecast for which we have data
fcst_model = "AROME_Arctic_prod", # the name of the deterministic model as in the file name
parameter = "T2m", # We are going to read 2m temperature
lead_time = seq(0, 48, 3), # We have data for lead times 0 - 48 at 3 hour intervals
by = "6h", # We have forecasts every 6 hours
file_path = here("data/vfld"), # We don't include AROME_Arctic_prod in the path...
file_template = "vfld", # ...because it's in the template
return_data = TRUE # We want to get some data back - by default nothing is returned
)
```
We don't have to only handle one model at a time, as long as they have the same file format and fit the same template. We also have data for MEPS, which is an ensemble, but we can just read member 0 by setting the correct template. When there are different options for each fcst_model they can be passed as a named list as below:
```{r message=FALSE}
read_forecast(
start_date = 2019021700,
end_date = 2019021718,
fcst_model = c("AROME_Arctic_prod", "MEPS_prod"),
parameter = "T2m",
lead_time = seq(0, 48, 3),
by = "6h",
file_path = here("data/vfld"),
file_template = list(
AROME_Arctic_prod = "vfld",
MEPS_prod = "{fcst_model}/vfld{fcst_model}mbr000{YYYY}{MM}{DD}{HH}{LDT2}"
),
return_data = TRUE
)
```
If we want to write the data to sqlite files we need to tell it a path to write the data to. If the directories do not exist, they will be created. Let's write the data to "data/FCTABLE". We use FCTABLE as the directory name since in harp we refer to the SQLite files created by this process as FCTABLE files. We tell `read_forecast` that we want to output the data to files by setting `output_file_opts` to something. For SQLite files we can use the `sqlite_opts` function to set the options we need - usually you only need to set the path to the data using the `path` argument. Let's get the vertical temperature profiles as well as the 2m temperature. You can see the parameter names that harp understands by running the function `show_harp_parameters`
```{r}
show_harp_parameters()
```
So the upper air parameter for temperture is T.
```{r message=FALSE, warning=FALSE}
read_forecast(
start_date = 2019021700,
end_date = 2019021718,
fcst_model = c("AROME_Arctic_prod", "MEPS_prod"),
parameter = c("T2m", "T"),
lead_time = seq(0, 48, 3),
by = "6h",
file_path = here("data/vfld"),
file_template = list(
AROME_Arctic_prod = "vfld",
MEPS_prod = "{fcst_model}/vfld{fcst_model}mbr000{YYYY}{MM}{DD}{HH}{LDT2}"
),
output_file_opts = sqlite_opts(path = here("data/FCTABLE/deterministic"))
)
```
The function tells you where the data were written to, but you can also look under the ./data/FCTABLE/deterministic directory and see for yourself. By default `read_forecast` uses the "fctable" template, but this can be changed using the `template` argument in `sqlite_opts`. However, you should always keep the parameter name in the template.
There are a number of options in `read_forecast` that can be explored by looking at the help page for the function.
Your turn:
* 2m temperature is, by default, corrected for the difference in elevation between the model and observation site. Create sqlite files for uncorrected 2m temperature for AROME_Arctic_prod and MEPS_prod (member 0)
* Create sqlite files for 10m wind speed and dew point temperature for the upper air for AROME_Arctic_prod and MEPS_prod (member 0)
Solutions
* Create sqlite files for uncorrected 2m temperature for AROME_Arctic_prod and MEPS_prod (member 0)
```{r message=FALSE, warning=FALSE}
read_forecast(
start_date = 2019021700,
end_date = 2019021718,
fcst_model = c("AROME_Arctic_prod", "MEPS_prod"),
parameter = "T2m",
lead_time = seq(0, 48, 3),
by = "6h",
file_path = here("data/vfld"),
file_template = list(
AROME_Arctic_prod = "vfld",
MEPS_prod = "{fcst_model}/vfld{fcst_model}mbr000{YYYY}{MM}{DD}{HH}{LDT2}"
),
output_file_opts = sqlite_opts(path = here("data/FCTABLE/deterministic")),
transformation_opts = interpolate_opts(keep_model_t2m = TRUE)
)
```
* Create sqlite files for 10m wind speed and dew point temperature for the upper air for AROME_Arctic_prod and MEPS_prod (member 0)
```{r message=FALSE, warning=FALSE}
read_forecast(
start_date = 2019021700,
end_date = 2019021718,
fcst_model = c("AROME_Arctic_prod", "MEPS_prod"),
parameter = c("S10m", "Td"),
lead_time = seq(0, 48, 3),
by = "6h",
file_path = here("data/vfld"),
file_template = list(
AROME_Arctic_prod = "vfld",
MEPS_prod = "{fcst_model}/vfld{fcst_model}mbr000{YYYY}{MM}{DD}{HH}{LDT2}"
),
output_file_opts = sqlite_opts(path = here("data/FCTABLE/deterministic"))
)
```