[Datahub] Wfs pagination in data fetcher #1126

AlitaBernachot · 2025-02-19T11:01:47Z

Description

This PR introduces pagination when reading data through a WFS service. This feature is implemented in a new reader: the WfsReader.

If the service supports pagination, the WfsReader will handle pagination, fetching api at each read(). If service version is lower than 2.0.0 (does not handle pagination) the WfsReader returns GeoJsonReader or GmlReader (works as previously).

Architectural changes

Added a new WfsReader.

Screenshots

None

Quality Assurance Checklist

Commit history is devoid of any merge commits and readable to facilitate reviews
If new logic ⚙️ is introduced: unit tests were added
If new user stories 🤏 are introduced: E2E tests were added
If new UI components 🕹️ are introduced: corresponding stories in Storybook were created
If breaking changes 🪚 are introduced: add the breaking change label
If bugs 🐞 are fixed: add the backport <release branch> label
The documentation website 📚 has received the love it deserves

github-actions · 2025-02-19T11:03:01Z

Affected libs: feature-dataviz,
Affected apps: metadata-editor,

🚀 Build and deploy storybook and demo on GitHub Pages
📦 Build and push affected docker images

github-actions · 2025-02-19T11:50:05Z

📷 Screenshots are here!

jahow

This looks really good! I'd like to test this first but I think your implementation makes a lot of sense, thanks!

jahow · 2025-02-19T19:32:35Z

libs/util/data-fetcher/src/lib/readers/wfs.ts

+    return this.getData().then(
+      (result) =>
+        ({
+          itemsCount: result.items.length,
+        }) as DatasetInfo
+    )


I think this is supposed to be the total amount of items in the featureType, you should be able to get this information using ogc-client:

const count = (await this.endpoint.getFeatureTypeFull(featureTypeName)).objectCount

Right! i have updated the code like you have suggested.

jahow · 2025-02-19T19:35:47Z

libs/util/data-fetcher/src/lib/readers/wfs.ts

+    if (Array.isArray(this.sort) && this.sort.length > 0) {
+      const finalUrl = new URL(url)
+      const sorts = this.sort
+        .map(
+          (fieldSort) => `${fieldSort[1]}+${fieldSort[0] === 'asc' ? 'A' : 'D'}`
+        )
+        .join(',')
+
+      finalUrl.searchParams.append('sortBy', sorts)
+      url = finalUrl.toString()
+    }


looking good! would be great to have this in ogc-client :)

FYI i now set the params directly on the url string as searchParams is encoding the "+"

coveralls · 2025-02-20T09:06:49Z

coverage: 86.134% (+1.7%) from 84.386%
when pulling 445fa3f on wfs-datafetcher
into a73b34b on main.

jahow

This looks great, thank you! I added a few comments for things that could be clarified/simplified. I tried this reader with #1120 and it seems to work OK.

jahow · 2025-02-21T09:54:32Z

libs/feature/dataviz/src/lib/service/data.service.ts

      return this.getDownloadUrlsFromWfs(link.url.toString(), link.name).pipe(
        switchMap((urls) => {


You can probably just call openDataset right away:

Suggested change

return this.getDownloadUrlsFromWfs(link.url.toString(), link.name).pipe(

switchMap((urls) => {

return from(openDataset(wfsUrlEndpoint, 'wfs', {

wfsUrlEndpoint,

featureType: link.name,

})

Thanks, code updated

jahow · 2025-02-21T11:23:00Z

libs/util/data-fetcher/src/lib/readers/wfs.ts

+    url: string,
+    wfsUrlEndpoint: string,


I'm not sure I understand why you need both of these. Only using url everywhere would make it much simpler right?

in the getDownloadUrlsFromWfs() the url Endpoint is proxified, this is why i have a wfsUrlEndpoint. Should i ignore the proxified url then? Agree it would be simpler.

Ok i have removed one, now there is only one wfsUrlEndpoint

jahow · 2025-02-21T11:24:10Z

libs/util/data-fetcher/src/lib/readers/wfs.ts

+        )
+        .join(',')
+
+      finalUrl.searchParams.append('sortBy', sorts)


Suggested change

finalUrl.searchParams.append('sortBy', sorts)

finalUrl.searchParams.append('SORTBY', sorts)

to comply with the WFS spec :)

updated, thx

jahow · 2025-02-21T11:26:27Z

libs/util/data-fetcher/src/lib/data-fetcher.ts

+  options?: {
+    namespace?: string
+    wfsVersion?: WfsVersion
+    wfsUrlEndpoint?: string


Suggested change

wfsUrlEndpoint?: string

wfsFeatureType?: string

This would be more appropriate/explicit instead of reusing namespace

ok done, i kept namespace for gml and added wfsFeatureType for wfs, is that what you have in mind?

jahow · 2025-02-21T11:26:40Z

libs/util/data-fetcher/src/lib/data-fetcher.ts

+        reader = await WfsReader.createReader(
+          url,
+          options.wfsUrlEndpoint,
+          options.namespace


Suggested change

options.namespace

options.wfsFeatureType

jahow · 2025-02-21T11:28:38Z

libs/util/data-fetcher/src/lib/headers.ts

@@ -5,7 +5,7 @@ export function parseHeaders(httpHeaders: Headers): DatasetHeaders {
  if (httpHeaders.has('Content-Type')) {
    result.mimeType = httpHeaders.get('Content-Type').split(';')[0]
    const supported =
-      SupportedTypes.filter(
+      SupportedTypes.filter((type) => type !== 'wfs').filter(


Maybe add a comment here? just to make sure that someone encountering this code understands what's going on. Thanks!

jahow

Looking good, thank you for the work!

jahow · 2025-02-21T15:25:13Z

The e2e failures do not seem related, but that's a lot of failing tests :/

jahow · 2025-02-21T22:54:44Z

Merging this PR as the failure is similar as the one on main

AlitaBernachot force-pushed the wfs-datafetcher branch from 8586ef2 to b37add0 Compare February 19, 2025 11:36

AlitaBernachot force-pushed the wfs-datafetcher branch 8 times, most recently from ed85830 to 255637d Compare February 19, 2025 17:35

AlitaBernachot marked this pull request as ready for review February 19, 2025 17:36

AlitaBernachot requested a review from jahow February 19, 2025 17:36

jahow reviewed Feb 19, 2025

View reviewed changes

AlitaBernachot force-pushed the wfs-datafetcher branch 3 times, most recently from 16b04f7 to 66a24b2 Compare February 20, 2025 09:00

jahow requested changes Feb 21, 2025

View reviewed changes

AlitaBernachot requested a review from jahow February 21, 2025 15:13

jahow approved these changes Feb 21, 2025

View reviewed changes

AlitaBernachot added 5 commits February 21, 2025 17:51

feat: wfs pagination

2d0c8fb

fix: use uppercases for compliance with WFS spec

2abaaf9

refactor: add code comment

bad44f9

fix: address pr comments, remove useless params

cd4da96

fix: dont encode SORTBY +A +D

445fa3f

AlitaBernachot force-pushed the wfs-datafetcher branch from 299471e to 445fa3f Compare February 21, 2025 17:49

jahow merged commit 096cd6e into main Feb 21, 2025
13 of 14 checks passed

jahow deleted the wfs-datafetcher branch February 21, 2025 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Datahub] Wfs pagination in data fetcher #1126

[Datahub] Wfs pagination in data fetcher #1126

AlitaBernachot commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

jahow left a comment

jahow Feb 19, 2025

AlitaBernachot Feb 20, 2025

jahow Feb 19, 2025

AlitaBernachot Feb 21, 2025

coveralls commented Feb 20, 2025 •

edited

Loading

jahow left a comment

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow Feb 21, 2025

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow Feb 21, 2025

AlitaBernachot Feb 21, 2025

jahow left a comment

jahow commented Feb 21, 2025

jahow commented Feb 21, 2025

		return this.getDownloadUrlsFromWfs(link.url.toString(), link.name).pipe(
		switchMap((urls) => {

	finalUrl.searchParams.append('sortBy', sorts)
	finalUrl.searchParams.append('SORTBY', sorts)

[Datahub] Wfs pagination in data fetcher #1126

[Datahub] Wfs pagination in data fetcher #1126

Conversation

AlitaBernachot commented Feb 19, 2025 • edited Loading

Description

Architectural changes

Screenshots

Quality Assurance Checklist

github-actions bot commented Feb 19, 2025 • edited Loading

github-actions bot commented Feb 19, 2025 • edited Loading

jahow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Feb 20, 2025 • edited Loading

jahow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jahow left a comment

Choose a reason for hiding this comment

jahow commented Feb 21, 2025

jahow commented Feb 21, 2025

AlitaBernachot commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

coveralls commented Feb 20, 2025 •

edited

Loading