Refactor code to obtain recent statistics from Weblate API

[hturner]

Recent Statistic.R queries the Weblate API to collate data on new translations and translations marked for edit.

This code is unnecessarily complicated, making it difficult to read and maintain.

This task is to refactor the code, in particular extracting code into functions that could be used in other scripts that query the Weblate API.

Improvements could include:

1. Structure code into clear, reusable functions, e.g.
   get_auth_handle(API_TOKEN): get handle to pass to curl_fetch_memory()
fetch_results(endpoint, handle): get results for a specific endpoint (all pages)
2. Instead of manually computing page counts, loop until the API returns no more results (length of returned results is zero). The results for each page can be saved in a list and combined after breaking from the loop.
3. Instead of processing each page of results, process the results for all pages together, after the results have been combined.
4. Replace loops with vectorized code where possible, e.g. L86-92 could use match() as in L153:155
5. Consider defining helper functions for repeated code, e.g. resolve_language_codes(codes, Language_Statistics) to make the code cleaner.

Since the script is run daily via GitHub actions, we want to minimize the (total) number of dependencies. So:

• Use base R functions to remove dependence on {stringr} (ref: Try not to depend on stringr #9).
• Avoid adding new dependencies in refactoring.

Put each function in a separate R file in the root of the repository, so that the functions can be used across scripts with

helpers <- list.files("R", pattern = "\\.R$", full.names = TRUE)
lapply(helpers, source)

roxygen-style documentation would be helpful.

PRs should be made to the main branch.

[jonocarroll]

I can't assign myself, but I'm interested in this task

[RWParsons]

also very happy to help on this one if you want to work together @jonocarroll 😄

[cynthiahqy]

@jonocarroll I'm keen to join on this too -- I'm in the breakout room now

[cynthiahqy]

@hturner is it possible to get examples of the json response objects -- e.g. for testing refactoring to remove stringr dependencies?

[jonocarroll]

There seems to be an assumption of 200 results per page, but the docs say 100... https://docs.weblate.org/en/weblate-5.11.4/api.html#api-rate-limiting

The units endpoint does not appear to take a page_size https://docs.weblate.org/en/weblate-5.11.4/api.html#units

Testing does show 200.

If pages do not contain 200 results the calculated number of pages will be incorrect.

[hturner]

@jonocarroll I think it's best to avoid this calculation altogether with something like:

results <- list()
page <- 1
repeat {
  url <- paste0(base_url, "&page=", page)
  data <- fetch_json(url, handle)
  if (length(data$results) == 0) break
  results[[page]] <- data$results
  page <- page + 1
}
changes <- do.call(rbind, results)

The number of results does depend on the endpoint!

[smwindecker]

commenting so I can be added