Skip to content

WorlddevelopmentIndicator - added logic to skip file that doesn't exist #1920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
niveditasing merged 3 commits into from
Mar 17, 2026
+13 −5
Merged

WorlddevelopmentIndicator - added logic to skip file that doesn't exist #1920

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
21f2f55
added logic to skip file that doesn't exist
niveditasing Mar 16, 2026
556a589
added logic to skip file that doesn't exist
niveditasing Mar 16, 2026
07273a1
modified code
niveditasing Mar 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions scripts/world_bank/wdi/worldbank.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,9 +255,11 @@ def read_worldbank(iso3166alpha3, mode):
if file.startswith("API"):
file_to_open = file
break
assert file_to_open is not None, \
"Failed to find data for" + iso3166alpha3

if file_to_open is None:
logging.warning(
'Failed to find data for %s in the downloaded ZIP. Skipping.',
iso3166alpha3)
return None
df = None
# Captures any text contained in double quotatations.
line_match = re.compile(r"\"([^\"]*)\"")
Expand Down Expand Up @@ -415,15 +417,21 @@ def download_indicator_data(worldbank_countries, indicator_codes, mode):
for index, country_code in enumerate(worldbank_countries['ISO3166Alpha3']):
country_df = read_worldbank(country_code, mode)

if country_df is None:
continue

# Remove unneccessary indicators.
country_df = country_df[country_df['IndicatorCode'].isin(
indicators_to_keep)]

# Map country codes to ISO.
country_df['ISO3166Alpha3'] = country_code

# Add new row to main datframe.
country_df_list.append(country_df)
# 3. Handle the empty list case OUTSIDE the loop
if not country_df_list:
logging.error("No data was downloaded for any country.")
# Return empty DF with expected columns to satisfy the rest of the pipeline
return pd.DataFrame(columns=['StatisticalVariable', 'Year', 'Value'])

worldbank_dataframe = pd.concat(country_df_list)
# Map indicator codes to unique Statistical Variable.
Expand Down
Loading