Windkraft in Bayern → Datensatz: 1000 lines

[denis-bz]

Clicking Windkraft in Bayern → Datensatz under https://marktstammdatenregister.dev/
gives a .csv with exactly 1000 lines--

SEE907461190045,Oberbiberg,,,2023-10-06,472,35,30.0,30.0,Kitekraft Demonstrator bis 30kW Oberbiberg,EEG952304848577
...
SEE978377618305,Hagenbüchach,10.769887,49.524045,2010-04-20,472,35,2000.0,2000.0,Bürgerwindrad      Trübenbronn-Hagenbüchach UG (hb) ＆ Co. KG,EEG995916352513

Odd that it's exactly 1000 ?

Would you have an example of reading the zenodo files in python pandas ?
Thanks,
cheers
-- denis

[curiousleo]

This query: Windkraft in Bayern.

It shows 969 rows both in the HTML UI and in the CSV.

However, the underlying software (Datasette) automatically cuts off results at 1000. There are ways to raise this limit, but large data exports isn't really what this interface is made for -- more for quick prototyping and drilling into things.

I'd be happy to provide an example of reading the Zenodo files in Python pandas!

If you just want to look at a single data table, you can load it from Parquet:

import pandas as pd

einheitwind = pd.read_parquet("einheitwind.parquet")
print(einheitwind[einheitwind["Ort"] == "Kronach"])

To join data from different tables, you can use the SQLite file:

import pandas as pd
import sqlite3

with sqlite3.connect("bnetza_mastr_2023-12-22.sqlite3") as con:
    result = pd.read_sql_query(
        """
WITH Rechtsformen AS (
  SELECT
    Rechtsform AS Rechtsform_Id,
    COUNT(*) AS Anzahl
  FROM
    Marktakteur
  GROUP BY
    Rechtsform
)
SELECT
  Katalogwert.Wert as Rechtsform,
  Anzahl
FROM
  Rechtsformen
  LEFT JOIN Katalogwert ON (Rechtsformen.Rechtsform_Id = Katalogwert.Id)
ORDER BY
  Anzahl DESC
""",
        con,
    )

print(result)

I think that in future, the combination of DuckDB (a faster database than SQLite) with Polars (a faster dataframe library compared to Pandas) will make for an even better combination.

Thanks for your question! I hope this helps, and I'll think think about adding something like this to the README or somewhere else prominent.

[denis-bz]

Hi Leonhard, thanks for your time and comments. I'm not much of a database guy, get along in pandas + numpy and the data I want is, for you, small -- 2.0G Gesamtdatenexport_20240731_24.1.zip 153M EinheitenWind.xml (36683, 72) 3.1M EinheitenWind.csv (28036, 13) -- the few columns I need There must be a market for faster DBs, DuckDB etc. but seems that a lot of geo people use xarray / geopandas -- imho clumsy / poor doc, speed is less important. (On pola-rs and hdf5 see the cartoon halfway down pola-rs/polars#3520 <pola-rs/polars#3520> (deleted 28 Sept: A question, EinheitenWind.xml is for onshore only -- my mistake, dropped AGS NaNs) Thanks, cheers -- denis

…

On 22 Sep 2024, at 10:34, Leonhard Markert ***@***.***> wrote: Hi! The usual site is unavailable right now unfortunately, working on it ... The backup site lives here: Windkraft in Bayern. It shows 969 rows both in the HTML UI and in the CSV. However, the underlying software (Datasette) automatically cuts off results at 1000. There are ways to raise this limit, but large data exports isn't really what this interface is made for -- more for quick prototyping and drilling into things. I'd be happy to provide an example of reading the Zenodo files in Python pandas! If you just want to look at a single data table, you can load it from Parquet: import pandas as pd einheitwind = pd.read_parquet("einheitwind.parquet") print(einheitwind[einheitwind["Ort"] == "Kronach"]) To join data from different tables, you can use the SQLite file: import pandas as pd import sqlite3 with sqlite3.connect("bnetza_mastr_2023-12-22.sqlite3") as con: result = pd.read_sql_query( """ WITH Rechtsformen AS ( SELECT Rechtsform AS Rechtsform_Id, COUNT(*) AS Anzahl FROM Marktakteur GROUP BY Rechtsform ) SELECT Katalogwert.Wert as Rechtsform, Anzahl FROM Rechtsformen LEFT JOIN Katalogwert ON (Rechtsformen.Rechtsform_Id = Katalogwert.Id) ORDER BY Anzahl DESC """, con, ) print(result) I think that in future, the combination of DuckDB (a faster database than SQLite) with Polars (a faster dataframe library compared to Pandas) will make for an even better combination. Thanks for your question! I hope this helps, and I'll think think about adding something like this to the README or somewhere else prominent.

[curiousleo]

Hi Denis,

I created this for you, perhaps it is a good starting point? https://observablehq.com/d/66bd1dbd44c01dbd
It queries and downloads the first 100 rows of the EinheitenWind table to display it (there's a "slice" menu at the bottom where you can change this, including downloading the entire table).
You can use the controls at the bottom to filter rows, select columns, sort the table, etc.
I think this is nice for exploration.

As for how you'll process the data in Python, I personally think DuckDB and Polars are great.

[denis-bz]

Hi Leonhard fyinfo, a few entries in EinheitenWind.csv from Gesamtdatenexport_20250601_25.1.zip have Nabenhoehe 0 or Rotordurchmesser 0 or both ? Also there’s a Nabenhoehe 300m (I haven’t checked newer .zips — takes a while to download when I only want EinheitenWind.csv) Thanks, cheers — denis

On 11 Nov 2024, at 15:30, Leonhard Markert ***@***.***> wrote: As for how you'll process the data in Python, I personally think DuckDB and Polars are great.

do you know if Polars can handle 3d, 4d arrays ?

[denis-bz]

Hi Marktstammdatenregister people, have you had a chance to look at this ? please let me know Thanks, cheers — denis

Begin forwarded message: From: denis-bz-py ***@***.***> Subject: Nabenhoehe 0, Rotordurchmesser 0 in EinheitenWind.csv ? Date: 16 November 2025 at 16:15:17 CET To: marktstammdatenregister-dev/mastr ***@***.***> Reply-To: denis-bz-py ***@***.***> Hi Leonhard fyinfo, a few entries in EinheitenWind.csv from Gesamtdatenexport_20250601_25.1.zip have Nabenhoehe 0 or Rotordurchmesser 0 or both ? Also there’s a Nabenhoehe 300m

also a GeplantesInbetriebnahmedatum 2040

…

(I haven’t checked newer .zips — takes a while to download when I only want EinheitenWind.csv) Thanks, cheers — denis