Skip to content

Commit 184eaac

Browse files
clamprCopilot
clampr
and
Copilot
authored
feat: add parquet data docs (#37)
* feat: add parquet data docs * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 8dcab2a commit 184eaac

19 files changed

Lines changed: 225 additions & 53 deletions

‎docs/data/_category_.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
{
22
"position": 2,
3-
"label": "Bulk Data",
3+
"label": "Data Access",
44
"collapsible": true,
55
"collapsed": true,
66
"link": {
77
"type": "generated-index",
8-
"title": "Bulk Data"
8+
"title": "Data Access"
99
}
1010
}

‎docs/data/bulk/_category_.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"position": 3,
3+
"label": "Bulk Data",
4+
"collapsible": true,
5+
"collapsed": true,
6+
"link": {
7+
"type": "doc",
8+
"id": "data-bulk-overview"
9+
}
10+
}

‎docs/data/bulk/daily.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
title: Daily Bulk Data | Data Access
3+
sidebar_label: Daily Data
4+
sidebar_position: 2
5+
---
6+
7+
# Daily Data
8+
9+
This endpoint provides **one Parquet file per year**.
10+
11+
:::warning[Beta]
12+
This interface is currently in beta. We are actively working on improving it and adding new features. The formats and data structures may change in the future. We recommend checking back regularly for updates and improvements.
13+
:::
14+
15+
## Endpoints
16+
17+
Annual data dumps, including model data as a substitute for missing observations, are available here:
18+
19+
```
20+
https://data.meteostat.net/daily/{year}.parquet
21+
```
22+
23+
Please replace `{year}` with the desired year number.
24+
25+
## Structure
26+
27+
Parquet files use a columnar storage format. The columns correspond to the parameter codes described [here](/parameters?g=daily&d=1).
28+
29+
More information on the data formats and weather condition codes is available [here](/formats).
30+
31+
:::tip[Geographical Coordinates]
32+
The Parquet files only contain the station ID, but not the geographical coordinates. To retrieve the coordinates, please download the [weather station database](/data/weather-stations) and join it based on the station ID.
33+
:::
34+
35+
### Data Sources
36+
37+
Each data column has a corresponding source column with the `_source` suffix.

‎docs/data/bulk/hourly.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
title: Hourly Bulk Data | Data Access
3+
sidebar_label: Hourly Data
4+
sidebar_position: 1
5+
---
6+
7+
# Hourly Data
8+
9+
This endpoint provides **one Parquet file per year**. The provided data is aggregated from historical databases, METAR reports and SYNOP data.
10+
11+
:::warning[Beta]
12+
This interface is currently in beta. We are actively working on improving it and adding new features. The formats and data structures may change in the future. We recommend checking back regularly for updates and improvements.
13+
:::
14+
15+
## Endpoints
16+
17+
Annual data dumps, including model data as a substitute for missing observations, are available here:
18+
19+
```
20+
https://data.meteostat.net/hourly/{year}.parquet
21+
```
22+
23+
Please replace `{year}` with the desired year number.
24+
25+
## Structure
26+
27+
Parquet files use a columnar storage format. The columns correspond to the parameter codes described [here](/parameters?g=hourly&d=1).
28+
29+
More information on the data formats and weather condition codes is available [here](/formats).
30+
31+
:::tip[Geographical Coordinates]
32+
The Parquet files only contain the station ID, but not the geographical coordinates. To retrieve the coordinates, please download the [weather station database](/data/weather-stations) and join it based on the station ID.
33+
:::
34+
35+
### Data Sources
36+
37+
Each data column has a corresponding source column with the `_source` suffix.

‎docs/data/bulk/monthly.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: Monthly Bulk Data | Data Access
3+
sidebar_label: Monthly Data
4+
sidebar_position: 3
5+
---
6+
7+
# Monthly Data
8+
9+
This endpoint provides **a single Parquet file** containing historical monthly data for all weather stations.
10+
11+
:::warning[Beta]
12+
This interface is currently in beta. We are actively working on improving it and adding new features. The formats and data structures may change in the future. We recommend checking back regularly for updates and improvements.
13+
:::
14+
15+
## Endpoints
16+
17+
The historical monthly data, including model data as a substitute for missing observations, for all weather stations is available here:
18+
19+
```
20+
https://data.meteostat.net/monthly.parquet
21+
```
22+
23+
## Structure
24+
25+
Parquet files use a columnar storage format. The columns correspond to the parameter codes described [here](/parameters?g=monthly&d=1).
26+
27+
More information on the data formats and weather condition codes is available [here](/formats).
28+
29+
:::tip[Geographical Coordinates]
30+
The Parquet files only contain the station ID, but not the geographical coordinates. To retrieve the coordinates, please download the [weather station database](/data/weather-stations) and join it based on the station ID.
31+
:::
32+
33+
### Data Sources
34+
35+
Each data column has a corresponding source column with the `_source` suffix.

‎docs/data/bulk/overview.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
title: Bulk Data | Data Access
3+
sidebar_label: Overview
4+
id: data-bulk-overview
5+
slug: /data/bulk
6+
sidebar_position: 1
7+
---
8+
9+
import DocCardList from '@theme/DocCardList';
10+
11+
# Bulk Data
12+
13+
The Meteostat bulk data interface provides access to weather and climate data across **all** weather stations in **Parquet** format. It's especially useful for large-scale, spatial analysis. Users are **not required to sign up** for this service.
14+
15+
:::warning[Beta]
16+
This interface is currently in beta. We are actively working on improving it and adding new features. The formats and data structures may change in the future. We recommend checking back regularly for updates and improvements.
17+
:::
18+
19+
## 🔌 Access
20+
21+
This interface does not require an API key. However, when using this service you must comply with our [terms of service](/terms). Please make sure to cache data if possible and forbear from sending malicious calls to this service.
22+
23+
Data is available in three different granularities:
24+
25+
- [Hourly data](/data/bulk/hourly)
26+
- [Daily data](/data/bulk/daily)
27+
- [Monthly data](/data/bulk/monthly)
28+
29+
:::warning
30+
Years without data will return an HTTP `404` status code.
31+
:::
32+
33+
## 🚀 Quick Start
34+
35+
The download of annual data dumps is dead simple and doesn’t even require an API key:
36+
37+
```bash
38+
curl "https://data.meteostat.net/hourly/2024.parquet" --output "2024.parquet"
39+
```
40+
41+
## 🔄 Update Cycle
42+
43+
The dumps are updated regularly, depending on the granularity of records. Recent hourly data should be available after a maximum of 24 hours.
44+
45+
## 👀 Learn More {#learn-more}
46+
47+
<DocCardList />

‎docs/data/overview.md‎

Lines changed: 3 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Bulk Data
2+
title: Data Access
33
sidebar_label: Overview
44
id: data-overview
55
slug: /data
@@ -8,35 +8,9 @@ sidebar_position: 1
88

99
import DocCardList from '@theme/DocCardList';
1010

11-
# Bulk Data
11+
# Data Access
1212

13-
The Meteostat bulk data interface provides access to full data dumps of individual weather stations. The data is provided in CSV (weather data) and JSON (weather stations) format. Users are **not required to sign up** for this service.
14-
15-
## 🔌 Access
16-
17-
```
18-
https://data.meteostat.net
19-
```
20-
21-
:::warning
22-
The dumps of weather stations which didn't report data for the requested frequency or data type will return an HTTP `404` status code.
23-
:::
24-
25-
In contrast to our JSON API the Bulk Data interface does not require an API key. However, when using this service you must comply with our [terms of service](/terms). Please make sure to cache data if possible and forbear from sending malicious calls to this service.
26-
27-
## 🚀 Quick Start
28-
29-
The download of annual data dumps is dead simple and doesn’t even require an API key:
30-
31-
```bash
32-
curl "https://data.meteostat.net/hourly/2024/10637.csv.gz" --output "10637-2024.csv.gz"
33-
```
34-
35-
With our [Python library](/python/) we're providing a simple, yet powerful, wrapper for bulk data dumps. If you're into more complex analysis you should definitely have a look at it.
36-
37-
## 🔄 Update Cycle
38-
39-
To keep the load on our infrastructure as low as possible, Meteostat updates bulk data dumps individually for each weather station. The dumps are updated regularly, depending on the granularity of records. Recent hourly data should be available after a maximum of 24 hours.
13+
Meteostat provides open and free access to historical weather and climate data. Users can download full [time series](/data/timeseries/) of individual weather stations provided in **CSV** format and [bulk data](/data/bulk/) in **Parquet** format. Weather station [meta data](/data/weather-stations) is provided in **JSON** and **SQL** formats. Users are **not required to sign up** for this service.
4014

4115
## 👀 Learn More
4216

‎docs/data/timeseries/daily.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Daily Data | Bulk Data
2+
title: Daily Data | Data Access
33
sidebar_label: Daily Data
44
sidebar_position: 2
55
---
66

77
# Daily Data
88

9-
This endpoint provides one GZ compressed CSV file per weather station and year.
9+
This endpoint provides one **gzip-compressed CSV file** per weather station and year.
1010

1111
## Endpoints
1212

@@ -16,11 +16,12 @@ Annual data dumps, including model data as substitute for missing observations,
1616
https://data.meteostat.net/daily/{year}/{station}.csv.gz
1717
```
1818

19-
Please replace `{year}` with the desired year number and `{station}` with the ID of a weather station.
19+
Please replace `{year}` with the desired year number and `{station}` with the ID of a [weather station](/data/weather-stations).
2020

2121
## Structure
2222

23-
CSV files provided through the Meteostat bulk data interface use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/formats#meteorological-parameters).
23+
CSV files use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/parameters?g=daily&d=1).
24+
The files **only contain the default set of parameters**. Additional parameters are not included in the data dumps, but can be retrieved through the [Python library](/python/).
2425

2526
More information on the data formats and weather condition codes is available [here](/formats).
2627

‎docs/data/timeseries/hourly.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Hourly Data | Bulk Data
2+
title: Hourly Data | Data Access
33
sidebar_label: Hourly Data
44
sidebar_position: 1
55
---
66

77
# Hourly Data
88

9-
This endpoint provides one GZ compressed CSV file per weather station and year. The provided data is being aggregated from historical databases, METAR reports and SYNOP data.
9+
This endpoint provides one **gzip-compressed CSV file** per weather station and year. The provided data is aggregated from historical databases, METAR reports and SYNOP data.
1010

1111
## Endpoints
1212

@@ -16,11 +16,12 @@ Annual data dumps, including model data as substitute for missing observations,
1616
https://data.meteostat.net/hourly/{year}/{station}.csv.gz
1717
```
1818

19-
Please replace `{year}` with the desired year number and `{station}` with the ID of a weather station.
19+
Please replace `{year}` with the desired year number and `{station}` with the ID of a [weather station](/data/weather-stations).
2020

2121
## Structure
2222

23-
CSV files provided through the Meteostat bulk data interface use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/formats#meteorological-parameters).
23+
CSV files use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/parameters?g=hourly&d=1).
24+
The files **only contain the default set of parameters**. Additional parameters are not included in the data dumps, but can be retrieved through the [Python library](/python/).
2425

2526
More information on the data formats and weather condition codes is available [here](/formats).
2627

‎docs/data/timeseries/monthly.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Monthly Data | Bulk Data
2+
title: Monthly Data | Data Access
33
sidebar_label: Monthly Data
44
sidebar_position: 3
55
---
66

77
# Monthly Data
88

9-
This endpoint provides one GZ compressed CSV file per weather station.
9+
This endpoint provides one **gzip-compressed CSV file** per weather station.
1010

1111
## Endpoints
1212

@@ -16,11 +16,12 @@ Data dumps, including model data as substitute for missing observations, are ava
1616
https://data.meteostat.net/monthly/{station}.csv.gz
1717
```
1818

19-
Please replace `{station}` with the ID of a weather station.
19+
Please replace `{station}` with the ID of a [weather station](/data/weather-stations).
2020

2121
## Structure
2222

23-
CSV files provided through the Meteostat bulk data interface use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/formats#meteorological-parameters).
23+
CSV files use commas as separators. Each file includes a header row containing the column names, which correspond to the parameter codes described [here](/parameters?g=monthly&d=1).
24+
The files **only contain the default set of parameters**. Additional parameters are not included in the data dumps, but can be retrieved through the [Python library](/python/).
2425

2526
More information on the data formats and weather condition codes is available [here](/formats).
2627

0 commit comments

Comments
 (0)