New glyph system

[michaelnmmeyer]

I finished setting this up.

The authority file is at project-documentation/DHARMA_glyphTaxonomy.xml.

The corresponding display is at https://dharmalekha.info/glyphs. The contents of the taxonomy was extracted from the data file I used so far. If you scroll down to the bottom of the Web page, you will see usage examples (and test data).

There is also a test file in test/DHARMA_INSTestGaiji.xml. The corresponding display is at https://dharmalekha.info/texts/INSTestGaiji. You can use it to see how the display behaves depending on the encoding of <g>.

Remaining problems are:

(1) I have not understood conventions regarding the contents of <g> and the mapping property from the taxonomy. As far as I'm concerned, what matters is that (1) the contents of <g> is the thing that should be displayed when the edition is rendered; (2) if the <g> is empty, the value of the corresponding mapping is displayed instead. This is how <g> is supposed to behave in TEI.

(2) I have set things up in such a way that both the old encoding and the new one remain valid, to ensure a smooth transition. But the use of <g type="pc"> creates a conflict: a symbol named pc is heavily used in Tamil inscriptions. I suggest we use <g type="punctuation"> to avoid the issue.

(3) I am very wary of applying global modifications to existing files, because (1) the potential for errors is massive; (2) people have many files that are not public yet and that the global update would ignore; (3) I can't preserve the formatting of the original file. This is why I have set things up in such a way that the old encoding and the new one can coexist. Backwards compatibility should be a primary goal at this stage.

(4) Something less important: the prefix "tax:" doesn't say anything about the type of data that is being referred to ("taxonomy" of what?), so "sym:" or "gly:", etc. would be a better fit.

[danbalogh]

Dear @michaelnmmeyer , a great many thanks for this and sorry about not responding sooner. I'm on leave this week, which is also why I haven't yet responded to your comments in the gdoc. I'll be able to give this more time early next week, when I'll consider your comments there and here carefully. Until then, a couple of quick replies.

In the authority file, we'll want a new set of tokens as described in the proposal. We could perhaps include the already existing tokens in a special subsection for deprecated tokens retained for backward compatibility. I'll give this more thought.

(2) I assume the Tamil folks have used pc for the pillaiyar culi symbol and you are right that we'd better change to something else. Your suggestion <g type="punctuation"> sounds fine to me; let's perhaps invite @arlogriffiths to give an opinion on this before making it final.
(3) I think we will nonetheless need to do a global replace in the existing published files. I've already started setting up a spreadsheet listing existing values and replacements; once the details are finalised, you and I can iron that out quickly.
(4) my preference would then be "sym:" because it's friendlier than glyph. @arlogriffiths , what do you think?

[arlogriffiths]

many thanks to @michaelnmmeyer from me as well.

(2) I agree with <g type="punctuation"> to replace <g type="pc">.

(4) I concur with preference for "sym:".
[daba edited because some of the code was not visible]

[danbalogh]

I think it's best if we move open discussion from the symbol management proposal gdoc to this thread. So here it is.

Topic: if feasible in the schema, values of @ref on <g> not beginning with “sym:” should be prohibited.
Michaël said: This is not desirable, because custom URL schemes like "tax", "bib", etc. are supposed to abbreviate "real" URLs. The URL "tax:danda" and "https://dharmalekha.info/glyphs/danda", for instance, should be treated the same.
Dan says now: I'm aiming at reducing human error because I imagine people will tend to forget the "sym:". If I omit the "bib:" from a bibliographic reference (e.g. <ptr target="Kielhorn1898-1899_01"/>), I get an error message saying "Bibliographic prefix is bib:". This still happens if I enter a URL as the value of @target. Why should the same not be done for symbol refs?

Topic: display of encoded g elements
Original proposal

• by default, display the content of the (the default, dedicated or arbitrary placeholder)
• the display of <g type=”numeral”> is to remain unchanged
• for all other @type values on <g>, do the following
• when @ref is present on the <g>, look up the token in the authority file and override the default display as follows
  • if the token is not found in the authority file, or if the token has neither <mapping> nor a <localProp> name, keep the default display (the content of the <g>)
    • [alternatively: display the token instead, bracketed in the same way as the localProp name below]
    • note, in principle none of these situations should occur, so this is basically just a fallback in case of encoding error
  • if the token has a localProp name AND no <mapping>, then display the @value of the <localProp name="name"> (ignoring the content of the <g>)
  • if the token has <mapping>, then display the content of <mapping> (ignoring the content of the <g> and localProp name)

Michaël said:
This encoding is backwards. A default value is something that should be used whenever the user omits to indicate a value. Why should the user indicate a custom value within if the text is going to be replaced with something else? Am I missing something?
For now, I do this. If the <g> element is not empty, its contents is displayed. If this fails (because <g> is empty), I display the "mapping" value from the authority file. If this fails (because there is no "mapping" in the authority file), I display the first "name" value from the authority file. If this fails (because there is no "name" in the authority file), I display the contents of the "type" attribute.
Dan says now:
You do seem to be missing important details. The order in which I described the details was "backwards" if you like, from default to preferred. But the desired prioritisation is, I think, clear from that description. As per the proposal, the content of the is not for "indicating a value" but a placeholder. It's the encoding and the reference to the taxonomy that is supposed to indicate what kind of g we are dealing with. The function of the placeholder is 1, to provide a fallback display when nothing else is encoded (or not encoded correctly); 2, (arbitrary placeholders) to allow people to use dingbats to their hearts' desire in non-XML contexts, and to keep those dingbats in XML files, only adding a g tag but not replacing the dingbat with a default placeholders. Users are never asked "to indicate a custom value within <g>", although they are allowed to use a custom character within that element when they absolutely want to. The priority for display, in decreasing order, is: 1, mapping; 2, name; 3, content. If you wish, displaying the type may be retained as a legacy solution for empty g elements without mapping and name
So, in decreasing order of priority, g element display should be handled as follows.

1. display the mapping. This is the preferred display.
2. if and only if mapping is not available (because no ref on the g, token not found, or no mapping for the token), display the name. This is for symbols for which we chose not to create a mapping equivalent.
3. if and only if neither mapping nor name are available (because no ref on the g, or token not found), display the content of the <g>. This is the fallback option for cases where somebody did not care to add a reference to the authority file, or when they used an incorrect token.
4. if and only if neither of the above three are available (because the g is empty), display the value of @type. This is the legacy option for symbols encoded with the old taxonomy, which I still think should be batch-converted to the new system.

[danbalogh]

I have finished populating the file with the initial set of new tokens. I am maintaining a list (spreadsheet) of how old tokens correspond to the new tokens, listing all of the old tokens that Michael harvested from our corpus last year and showing which new token it should be replaced with. I will share that list once we have come to a final agreement on the new tokens, which I'd like @arlogriffiths to skim through and comment. He can do it either directly in the file or I can send him the current work-in-progress version of the spreadsheet.
The encoding system set up by Michael seems to be fine (aside from the eventual image concerns now in #425), but I have not been able to test how the content added by me looks. How long does it take for a git push of this file to appear on Dharmalekha? I did an experimental push this morning but it still hasn't appeared on the website (much less the push I did a few minutes ago).
In the near future I'll want to add some instructions for editing the file, and it may also be a good idea to add in some form the list of correspondences between old and new tokens.
As I've also noted in the file, I think we can simply discard the list of old tokens which Michael added to the first version of the authority file, because the old tokens are currently in @type and should never be used in @ref.

[danbalogh]

I am done with the table of new glyph tokens, with a correspondence list to existing tokens harvested earlier from the corpus by Michael. The table is here. Columns A-D concern batch replacement; columns E-H concern the contents of the new authority file. What I need you to do, as soon as possible (say, by next Monday?), is the following:

1. @arlogriffiths and @manufrancis : look over the tokens (column B), the full (but short) glyph names (column E), the proposed Unicode symbol mapping (column F) and the glyph descriptions (column G). If there is anything you are unhappy with, add a positive and constructive comment saying exactly what you want changed to what. Edit the table only where really necessary (correcting typos, adding new content) and better just comment when existing content needs a change.
2. @manufrancis : in particular, look at the Tamil glyphs (marked with the text "CHECK @emfr"). I've created the list of tokens based on existing tokens found in the corpus, but some may be duplicates as I can't be sure whether some existing tokens are meant to be abbreviations/alternative spellings or separate glyphs (e.g. nel and nellu, muntiri and muntirikai). I may also have erred on the other side, merging two different signs into one. So please check columns A (old token) and B (new token) and make sure there is only one new token for each symbol (i.e. put the same new token next to both old tokens). Check also the properly transliterated glyph names. Add a mapping character if you want one. If possible, add a description. If you do that now in the sheet, I'll insert it into the XML file. If you can't do it now, you'll need to do it later, directly in the XML.
3. @michaelnmmeyer : don't start any batch refactoring/replacement, but please look at the revised refactoring process in the glyph management gdoc in combination with the above spreadsheet for the old and new values. Comment in the Google Doc if you see a better way to get the desired result.
4. @arlogriffiths , if you want to add exotic stuff to the authority file, such as complex Balinese punctuation marks, I propose we leave it for later and do it directly in the XML once the system is up and working.

[manufrancis]

Done as far as I am concerned.

[danbalogh]

Great. I'll pipe up if there is anything else.

[danbalogh]

@manufrancis , thanks for doing these directly in the XML, it saves me a lot of work. There are still several entries that have no mapping (e.g. kalam, kalancu, kannaru, etc.). Are these yet to be done? No urgency if yes, I just wanted to know. Or is it that you don't want mapping for these (i.e. display the name instead)? In that case I'll delete the <!--CHECK @EMFR--> comments under mapping or you can do it yourself if you are willing.

[manufrancis]

There is no mapping when there is no unicode character code yet available. But I guess some will soon be available. I will add them when available. The display of the glyph's name (column E) is indeed wanted. I will delete <!--CHECK @EMFR--> in the authority file.

[manufrancis]

Done with deleting in the authority file for missing mapping.