Code

Harvest dataservices#3029

Code

Merged

ThibaudDauce merged 72 commits intomasteropendatateam/udata:masterfrom harvest_dataservicesopendatateam/udata:harvest_dataservicesCopy head branch name to clipboardJun 14, 2024

+408-143

Commits

Commits on Apr 30, 2024

• Switch DCAT backend to not use one job for each dataset

  ThibaudDauce

  committed
  9b35b43

  Copy full SHA for 9b35b43

• Fix missing owner/org in new datasets

  ThibaudDauce

  committed
  cb622a8

  Copy full SHA for cb622a8

• Remove prints

  ThibaudDauce

  committed
  cdd5f01

  Copy full SHA for cdd5f01

• Refactor using two functions

  ThibaudDauce

  committed
  35ad714

  Copy full SHA for 35ad714

• Add back should_stop

  ThibaudDauce

  committed
  76d99d8

  Copy full SHA for 76d99d8

• Add back autoarchive and done with failed items

  ThibaudDauce

  committed
  e90ddcc

  Copy full SHA for e90ddcc

• Merge branch 'master' into sync_harvest_backend

  ThibaudDauce

  authored
  203b394

  Copy full SHA for 203b394

Commits on May 2, 2024

• Always returns the graphs for debug

  ThibaudDauce

  committed
  5dd3c04

  Copy full SHA for 5dd3c04

• Add test for stopping due to HARVEST_MAX_ITEMS

  ThibaudDauce

  committed
  d8eaf45

  Copy full SHA for d8eaf45

Commits on May 14, 2024

• Update test backends

  ThibaudDauce

  committed
  ebf2af7

  Copy full SHA for ebf2af7

• Fix some tests

  ThibaudDauce

  committed
  a2701af

  Copy full SHA for a2701af

• Revert ID change for FakeBackend

  ThibaudDauce

  committed
  a163b70

  Copy full SHA for a163b70

• Simplify SyncBackend

  ThibaudDauce

  committed
  3693bd2

  Copy full SHA for 3693bd2

Commits on May 21, 2024

• Merge branch 'master' into sync_harvest_backend

  maudetes

  authored
  d5c0e29

  Copy full SHA for d5c0e29

• fix wrong remote_id

  ThibaudDauce

  committed
  a71b46e

  Copy full SHA for a71b46e

Commits on May 23, 2024

• Merge branch 'master' into sync_harvest_backend

  ThibaudDauce

  authored
  914b69d

  Copy full SHA for 914b69d

• Remove dead code

  ThibaudDauce

  committed
  fc98e28

  Copy full SHA for fc98e28

• update comment

  ThibaudDauce

  committed
  20ce7e4

  Copy full SHA for 20ce7e4

Commits on May 27, 2024

• Merge branch 'master' into sync_harvest_backend

  ThibaudDauce

  authored
  c3c4c27

  Copy full SHA for c3c4c27

• Move docstring

  ThibaudDauce

  committed
  b9b41e2

  Copy full SHA for b9b41e2

• Switch is_done() do its own function

  ThibaudDauce

  committed
  0a6dfa5

  Copy full SHA for 0a6dfa5

• Rename process_datasets method

  ThibaudDauce

  committed
  523c754

  Copy full SHA for 523c754

• yield instead of callback

  ThibaudDauce

  committed
  064f8eb

  Copy full SHA for 064f8eb

• fix other backends

  ThibaudDauce

  committed
  263c3a3

  Copy full SHA for 263c3a3

Commits on May 29, 2024

• Merge branch 'master' into sync_harvest_backend

  ThibaudDauce

  committed
  6b6d11e

  Copy full SHA for 6b6d11e

• Update changelog

  ThibaudDauce

  committed
  9124fc0

  Copy full SHA for 9124fc0

• Remove unused process method

  ThibaudDauce

  committed
  e9b9af1

  Copy full SHA for e9b9af1

Commits on May 30, 2024

• Skip dataset without remote_id

  ThibaudDauce

  committed
  318603d

  Copy full SHA for 318603d

• Harvest dataservices

  ThibaudDauce

  committed
  e1b1bc0

  Copy full SHA for e1b1bc0

• Add more data inside dataservices

  ThibaudDauce

  committed
  d536874

  Copy full SHA for d536874

• Revert hvd labels from rebase

  ThibaudDauce

  committed
  ce30a60

  Copy full SHA for ce30a60

• Fix tests

  ThibaudDauce

  committed
  3e4d901

  Copy full SHA for 3e4d901

• Remove duplicate code

  ThibaudDauce

  committed
  ae02115

  Copy full SHA for ae02115

• wip add printing graphs

  ThibaudDauce

  committed
  345f05b

  Copy full SHA for 345f05b

• Remove duplicate harvest update info

  ThibaudDauce

  committed
  735f6d5

  Copy full SHA for 735f6d5

• Add link between dataservices and datasets

  ThibaudDauce

  committed
  b83d697

  Copy full SHA for b83d697

• link dataset with dataservices

  ThibaudDauce

  committed
  29cede6

  Copy full SHA for 29cede6

Commits on Jun 3, 2024

• Use forked XSLT

  ThibaudDauce

  committed
  2c7a24d

  Copy full SHA for 2c7a24d

Commits on Jun 4, 2024

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  committed
  23d8111

  Copy full SHA for 23d8111

• fix merge issues

  ThibaudDauce

  committed
  aa125aa

  Copy full SHA for aa125aa

• Fix tests

  ThibaudDauce

  committed
  d667b7b

  Copy full SHA for d667b7b

• Improve some skips

  ThibaudDauce

  committed
  e6816c9

  Copy full SHA for e6816c9

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  committed
  f630a25

  Copy full SHA for f630a25

• Remove merge tag

  ThibaudDauce

  committed
  4eec493

  Copy full SHA for 4eec493

Commits on Jun 5, 2024

• Separate metadata

  ThibaudDauce

  committed
  f5c6917

  Copy full SHA for f5c6917

• Add comment

  ThibaudDauce

  committed
  c9b9940

  Copy full SHA for c9b9940

• Add dataservice card to harvester admin

  ThibaudDauce

  committed
  69ec784

  Copy full SHA for 69ec784

• Revert XSLT modifications

  ThibaudDauce

  committed
  3403944

  Copy full SHA for 3403944

• Remove some blanks lines

  ThibaudDauce

  committed
  7c5b0ca

  Copy full SHA for 7c5b0ca

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  authored
  f5a5108

  Copy full SHA for f5a5108

• Update changelog

  ThibaudDauce

  committed
  8e3e5ef

  Copy full SHA for 8e3e5ef

• Do not duplicate datasets on each harvesting

  ThibaudDauce

  committed
  9a9a43b

  Copy full SHA for 9a9a43b

Commits on Jun 10, 2024

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  committed
  e94a23f

  Copy full SHA for e94a23f

• Cleanup imports

  ThibaudDauce

  committed
  5fee125

  Copy full SHA for 5fee125

• Fix changelog

  ThibaudDauce

  committed
  31bfda7

  Copy full SHA for 31bfda7

• Fix merge

  ThibaudDauce

  committed
  f477ef6

  Copy full SHA for f477ef6

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  authored
  8f910e6

  Copy full SHA for 8f910e6

Commits on Jun 11, 2024

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  authored
  aba27e4

  Copy full SHA for aba27e4

• Apply suggestions from code review

  Show description for 9f9a7f8
  

  ThibaudDauce

  and

  maudetes

  authored
  9f9a7f8

  Copy full SHA for 9f9a7f8

• add harvest metadata to API

  ThibaudDauce

  committed
  2a0da24

  Copy full SHA for 2a0da24

• harvest metadata as readonly

  ThibaudDauce

  committed
  dd54158

  Copy full SHA for dd54158

• Rename last_harvested_at and add harvest.created_at

  ThibaudDauce

  committed
  30ba48d

  Copy full SHA for 30ba48d

• Fix wrong attribute

  ThibaudDauce

  committed
  b083c37

  Copy full SHA for b083c37

• Do not empty the datasets list if no datasets found in harvesting

  ThibaudDauce

  committed
  fbe6f22

  Copy full SHA for fbe6f22

• fix casing

  ThibaudDauce

  committed
  7863547

  Copy full SHA for 7863547

• Fix tests

  ThibaudDauce

  committed
  b401700

  Copy full SHA for b401700

Commits on Jun 12, 2024

• Save node id if it's an URL

  ThibaudDauce

  committed
  1d6714a

  Copy full SHA for 1d6714a

Commits on Jun 13, 2024

• Add landing page as remote_url

  ThibaudDauce

  committed
  cc6b5e8

  Copy full SHA for cc6b5e8

• Merge branch 'master' into harvest_dataservices

  ThibaudDauce

  authored
  dc1a093

  Copy full SHA for dc1a093

Commits on Jun 14, 2024

• Rename rdf_node_id_as_url to follow the same name as dataset :-(

  ThibaudDauce

  committed
  d484199

  Copy full SHA for d484199

• Remove dynamic to harvest dataservice metadata

  ThibaudDauce

  committed
  fa816cc

  Copy full SHA for fa816cc

• Update udata/core/dataservices/models.py

  Show description for 546c1be
  

  ThibaudDauce

  and

  maudetes

  authored
  546c1be

  Copy full SHA for 546c1be