followup from "Faker-ruby has too many generators to maintain" faker-ruby/faker#2689

[stoivo]

Hi, I want to ask a few questions related to you comment, faker-ruby/faker#2689 (comment). Since faker uses I18n it supports translations with fallback to english I think? how to you know which yaml files you have to parse? since you only parse yaml files on demand?

[psibi]

For fakedata, you pass the locale explicitly. By default, it uses english. You can chose to use any other local explicitly, if you opt to it.

how to you know which yaml files you have to parse? since you only parse yaml files on demand?

Since we know the locale and the yaml files, based on the usage - you can parse the appropriate yaml file and maintain the cache of the parsed yaml file. So throughout the run of the program, you don't have to read the same yaml file again.

[stoivo]

I assume you use the filename to guess the scope of the file. So the file containting this yaml should be called valleyball.yml. I did some sampels and it seems to be done like this all the time. I don't think this is required for the current solution faker uses. I think I18n don't care file name.

en:
  faker:
    volleyball:
      team:
      - A.S.D. Pallavolo Torino
      - ACH Volley Ljubljana
      - ADC Pirelli
      - AERO Odolena Voda

[stoivo]

Aha, these where the lines I was interested in. https://github.com/fakedata-haskell/fakedata/blob/master/src/Config.hs#L428-L437

So If I choose to split the ja translations into multiple files (I can see this happening since is it a big file) you would need to change the code to handle it.

[psibi]

So If I choose to split the ja translations into multiple files (I can see this happening since is it a big file)

Which file you are thinking about splitting ? Right now, I see that all the translations are already split: https://github.com/faker-ruby/faker/tree/main/lib/locales/ja

[stoivo]

I missed something. My point was that split the nb translations this code would need to change.

For the japanize translations address.yml is 2.4MB. So we might split out the postal codes which is a big part of it into ja/address/postral_code.yml. Might. Like, I want to speed up ruby faker and this might be one of the things I will do in that process, I need to get the approval from the ruby maintainers.

[psibi]

My point was that split the nb translations this code would need to change.

Ah, I see. I would like to avoid such breakages if possible. :-)

For the japanize translations address.yml is 2.4MB. So we might split out the postal codes which is a big part of it into ja/address/postral_code.yml.

It would be interesting to see the numbers. How long does it take to parse the entire 2.4 MB vs the subset of it ? I haven't looked the Ruby code in detail - but how does it do the parsing now ? Does it do on demand and cache it ?

[stoivo]

in my mac, parsing 2.4 MB took ~400ms and some smaller files on ~10KB took 5ms. So it a huge benefit of only parsing whats needed. The curretn ruby implementation loads and parses all yaml files inside locate. All of them when asking for the first. I am quite sertain that it dosn't care what the files are named. It just merges all the content into one hash. I opened an issue to ask if a PR changing this behaviour would be approved. Still waiting for a response :). right now I am leaning in the direction of changing the file structure so I know what the file containt based on the filename.