Count column values may be incorrect

[Ortham]

As @Sharlikran has stated here, TES5Edit treats everything as optional unless it's always absolutely required.

As such, its source code isn't a reliable indicator of whether fields are required or not. Instead, the data found here can be used to determine that. (@Sharlikran, is there any link to the utility that generated that file?)

Also, I'm not sure if there's a field that needs to exist but can be repeated, so maybe it would be better to change * to mean "zero or more repeats". Looking through the stats file, there are 101 mentions of required fields that appear more than once.

[Ortham]

Also @Sharlikran, are there any stats available on which sub-fields are required in each field? Or do fields always require all their sub-fields? I'd suspect the latter, because otherwise you can't really tell what sub-fields are present, but I thought I'd ask.

[Sharlikran]

The program that made those stats is TESDumpStats by Lojack. It's a derivative of Dave Humphry's TES5Load. TESDumpStats will work with any Bethesda game except probably Morrowind. It is rudimentary in it's design only providing the needed stats.

[Sharlikran]

In reference to required fields, only records reference xEdit's aRequired argument. Elements within the struct cannot. However, many times records in the original release of a Bethesda game, in Fallout3.esm for example have less data. xEdit uses the argument aOptionalFromElement to handle the difference in the data size of the original release and what the updated GECK or CK saves.

[Sharlikran]

The file "FO3 Record Stats.7z" mentioned above contains both separate stats for each of the official DLC files for FO3 and an aggregate summary of the stats starting at "Combined stats:".

[Sharlikran]

For the sake of the project, how do you want to handle the 'Count' field? Would you like to define all the records first, then go back and correct the 'Count' field after deciphering xEdit's code? Just so you aren't drinking out a fire hose of information, so to speak.

[Ortham]

The file "FO3 Record Stats.7z" mentioned above contains both separate stats for each of the official DLC files for FO3 and an aggregate summary of the stats starting at "Combined stats:".

Ah, I assumed there was only one plugin in it.

For the sake of the project, how do you want to handle the 'Count' field? Would you like to define all the records first, then go back and correct the 'Count' field after deciphering xEdit's code? Just so you aren't drinking out a fire hose of information, so to speak.

I'll use the plugin stats to decide what fields are required. I'll probably leave that column blank in any new pages I create though, and go back to it later, as you say. I still don't get 'optional' sub-field data, but then I haven't taken a look at aOptionalFromElement yet either.

[Sharlikran]

The difference is that the argument aRequired applies to the entire subrecord. The argument aOptionalFromElement handles the byte difference in records when Bethesda changes them. CBash, xEdit when looking for 36 bytes would error and say, "Found 24 bytes expected 36." That argument is only to handle the difference in bytes of records for post release DLC or GECK updates.

My Wiki rough draft is here and I show a code example.

[Ortham]

Hmm, how do you determine which of the sub-fields those bytes you do have correspond to? Or, equivalently, how do you determine which sub-fields are missing? Do you just assume that the missing data came from the end of the field, or is there some validation program / code that checks for sensible values?

[Sharlikran]

Normally the sub-fields and values line up exactly. This is normally determined when running xDump for it's respective game since xDump will report the byte difference. Once it's decoded then you don't have to revise it. Also, although it's annoying when Bethesda makes that kind of change, they do at least append to the record for compatibility.

Elminster must have discovered that early on since it is consistent with all games xEdit supports. They all have cases where the record size has changed. This is also evident in all versions of Wrye since they all have at least some custom unpackers that are checking for the size of the record. Wrye also assumes that the missing sub-fields were appended.

[Sharlikran]

Oh I didn't really answer this either. Once the record is decoded if you found 24 bytes, you just count the bytes by looking at the definition to determine which fields make 24 bytes. Then the rest are then optional. Again saying optional is confusing because this is really just indicating a potential byte difference.

Normally it only occurs in the primary ESM like Fallout3.esm, or Skyrim.esm, subsequent DLC like Update.esm or BrokenSteel.esm for FO3 probably has the larger record and it's then overwriting the previous values. Those post release DLC files have the larger sized record with 36 bytes.

[Ortham]

Wrye also assumes that the missing sub-fields were appended.

It's a sensible assumption to make, if I were to update a format, I wouldn't go sticking new stuff in the middle, that wouldn't make any sense.

So what you're saying is that when we say that a sub-field is "optional", we actually mean "this sub-field wasn't present in the initial release, but got added at a later date, so may not appear in all plugins"?

[Sharlikran]

Correct.

Typically It appears to come from the in house editor that Bethesda uses which is why it's primarily in the game's main ESM File. However, Bethesda may not have updated the size for the inaugural public release of the GECK or CK which would have saved plugins with the older, smaller format.

[Sharlikran]

So that is the difference between aRequired and aOptionalFromElement. The argument aRequired applies to the subrecord, the argument aOptionalFromElement applies to the elements within the subrecord when applicable.