No, let's not tokenize hyphens and dashes

[ebeshero]

The collateX software does not normally tokenize hyphens and dashes, and we wondered last fall whether tokenizing them might improve alignments. We introduced a function in the Python script to insert newline characters around all forms of hyphens and dashes so they would be treated as their own tokens. However, in testing collation outputs over the holiday break, I took a moment to compare outputs with and without the function in operation, and discovered that our function is not helping us as much as we had hoped. In many cases the tokenized hyphens are not a problem, but they do create problems and alignment issues. NOT treating them as separate tokens creates longer aligned readings and a clearer view of variation. So we are really better to let collateX attach the hyphens and dashes to the word-token to their left (like it handles other punctuation). Here are examples of outputs with and without the function:

Python script APPLIES function to tokenize hyphens and dashes: output example from C-13:

<app>
		<rdgGrp n="['or', 'confine', 'a', 'mountain']">
			<rdg wit="f1818">or confine a mountain </rdg>
			<rdg wit="f1823">or confine a mountain </rdg>
			<rdg wit="fThomas">or confine a mountain </rdg>
			<rdg wit="f1831">or confine a mountain </rdg>
			<rdg wit="fMS">or confine a &lt;lb n=&quot;c56-0086__main__27&quot;/&gt;mountain </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['-']">
			<rdg wit="f1818">- </rdg>
			<rdg wit="f1823">- </rdg>
			<rdg wit="fThomas">- </rdg>
			<rdg wit="f1831">- </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['stream', 'with', 'a']">
			<rdg wit="f1818">stream with a </rdg>
			<rdg wit="f1823">stream with a </rdg>
			<rdg wit="fThomas">stream with a </rdg>
			<rdg wit="f1831">stream with a </rdg>
			<rdg wit="fMS">stream with a </rdg>
		</rdgGrp>
	</app>

Python script REMOVES function to tokenize hyphens and dashes: output example of the same passage from C-13:

<app>
		<rdgGrp n="['or', 'confine', 'a']">
			<rdg wit="f1818">or confine a </rdg>
			<rdg wit="f1823">or confine a </rdg>
			<rdg wit="fThomas">or confine a </rdg>
			<rdg wit="f1831">or confine a </rdg>
			<rdg wit="fMS">or confine a </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['mountain', 'stream']">
			<rdg wit="fMS">&lt;lb n=&quot;c56-0086__main__27&quot;/&gt;mountain stream </rdg>
		</rdgGrp>
		<rdgGrp n="['mountain-stream']">
			<rdg wit="f1818">mountain-stream </rdg>
			<rdg wit="f1823">mountain-stream </rdg>
			<rdg wit="fThomas">mountain-stream </rdg>
			<rdg wit="f1831">mountain-stream </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['with', 'a']">
			<rdg wit="f1818">with a </rdg>
			<rdg wit="f1823">with a </rdg>
			<rdg wit="fThomas">with a </rdg>
			<rdg wit="f1831">with a </rdg>
			<rdg wit="fMS">with a </rdg>
		</rdgGrp>
	</app>

The second alignment is better. Why?

• The first example created an <app> only around the - character and did not include all the witnesses, so it was not so easy to notice where the hyphen was absent. In the Variorum Viewer, this would be difficult to read.
• The second example shows the variation more clearly: some witnesses give us "mountain-stream" while one gives us just "mountain stream": that's the real difference and all the witnesses are present in the <app> to show it. That means that all witnesses will equally be able to show how they are different from each other (no one is left out of the app).
  @Yuying-Jin

[ebeshero]

@Yuying-Jin Here is a clearer example of the problem, from C-13:

Python script tokenizes hyphens and dashes:

<app>
		<rdgGrp n="['from']">
			<rdg wit="f1831">from </rdg>
		</rdgGrp>
		<rdgGrp n="['to', 'the', 'east']">
			<rdg wit="f1818">to the east </rdg>
			<rdg wit="f1823">to the east </rdg>
			<rdg wit="fThomas">to the east </rdg>
			<rdg wit="fMS">&lt;lb n=&quot;c56-0081__left_margin__3&quot;/&gt;to the east&lt;lb n=&quot;c56-0081__left_margin__4&quot;/&gt; </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['-', 'ward']">
			<rdg wit="fMS">- ward </rdg>
		</rdgGrp>
	</app>

Python script does NOT tokenize hyphens and dashes, same passage:

	<app>
		<rdgGrp n="['from']">
			<rdg wit="f1831">from </rdg>
		</rdgGrp>
		<rdgGrp n="['to', 'the']">
			<rdg wit="f1818">to the </rdg>
			<rdg wit="f1823">to the </rdg>
			<rdg wit="fThomas">to the </rdg>
			<rdg wit="fMS">&lt;lb n=&quot;c56-0081__left_margin__3&quot;/&gt;to the </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['east']">
			<rdg wit="f1818">east </rdg>
			<rdg wit="f1823">east </rdg>
			<rdg wit="fThomas">east </rdg>
		</rdgGrp>
		<rdgGrp n="['east-ward']">
			<rdg wit="fMS">east&lt;lb n=&quot;c56-0081__left_margin__4&quot;/&gt;-ward </rdg>
		</rdgGrp>
	</app>

[ebeshero]

One more example from C-13: This might have been the passage that motivated our attempt to tokenize hyphens and dashes in the first place.

Python script tokenizes hyphens and dashes:

<app>
		<rdgGrp n="['the']">
			<rdg wit="fMS">the </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['mont']">
			<rdg wit="f1818">Mont </rdg>
			<rdg wit="f1823">Mont </rdg>
			<rdg wit="fThomas">Mont </rdg>
			<rdg wit="f1831">Mont </rdg>
			<rdg wit="fMS">Mont </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['-', 'salêve;']">
			<rdg wit="fMS">- Salêve; </rdg>
		</rdgGrp>
		<rdgGrp n="['salêve,', 'a', 'hill', 'that', 'bounds', 'plainpalais', 'on', 'the', 'south.']">
			<rdg wit="f1818">Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="f1823">Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="fThomas">Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="f1831">Salêve, a hill that bounds Plainpalais on the south. </rdg>
		</rdgGrp>
	</app>
<app>
		<rdgGrp n="['he', 'soon', 'reached', 'the']">
			<rdg wit="f1818">He soon reached the </rdg>
			<rdg wit="f1823">He soon reached the </rdg>
			<rdg wit="fThomas">He soon reached the </rdg>
			<rdg wit="f1831">He soon reached the </rdg>
			<rdg wit="fMS">he soon reached &lt;lb n=&quot;c56-0083__main__13&quot;/&gt;the </rdg>
		</rdgGrp>
	</app>

Python script does NOT tokenize hyphens or dashes, and we apply longToken to msColl

(We apply <longToken> here to deal with the word "the" as well as the hyphenated name of the mountain: this corrects the alignment and reduces the number of <app> elements.)

<app>
		<rdgGrp n="['mont', 'salêve,', 'a', 'hill', 'that', 'bounds', 'plainpalais', 'on', 'the', 'south.']">
			<rdg wit="f1818">Mont Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="f1823">Mont Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="fThomas">Mont Salêve, a hill that bounds Plainpalais on the south. </rdg>
			<rdg wit="f1831">Mont Salêve, a hill that bounds Plainpalais on the south. </rdg>
		</rdgGrp>
		<rdgGrp n="['the mont-salêve;']">
			<rdg wit="fMS">&lt;longToken&gt;the Mont-Salêve;&lt;/longToken&gt; </rdg>
		</rdgGrp>
	</app>
<app>
		<rdgGrp n="['he', 'soon', 'reached', 'the']">
			<rdg wit="f1818">He soon reached the </rdg>
			<rdg wit="f1823">He soon reached the </rdg>
			<rdg wit="fThomas">He soon reached the </rdg>
			<rdg wit="f1831">He soon reached the </rdg>
			<rdg wit="fMS">he soon reached &lt;lb n=&quot;c56-0083__main__13&quot;/&gt;the </rdg>
		</rdgGrp>
	</app>

[Yuying-Jin]

@ebeshero Agreed: removing function to tokenize hyphens and dashes.
I find why we use this function at the time (the codes shown below). At the time, I wanted to move i never after the hyphen in the second line down. However, longToken did not work for it, so I tried to process hyphens. At present, I think this kind of misalignment can be ignored, because this misalignment is not obvious, while tokenizing hyphens and dashes causes many other obvious problems.

Tokenizing hyphens and dashes: output example from C-14:

<app>
		<rdgGrp n="['must die –', 'you', 'my companion, my playfellow,', '', 'my more than sister—', 'die', '–']">
			<rdg wit="fMS">&lt;longToken&gt;must die –&lt;/longToken&gt; &lt;lb n=&quot;c56-0103__main__17&quot;/&gt;you &lt;longToken&gt;my companion, my playfellow,&lt;/longToken&gt; &lt;lb n=&quot;c56-0103__main__18&quot;/&gt; &lt;longToken&gt;my more than sister—&lt;/longToken&gt; die – </rdg>
		</rdgGrp>
		<rdgGrp n="['must die;', 'you,', 'my', 'playfellow,', 'my', 'companion,', 'my more than sister.']">
			<rdg wit="f1818">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
			<rdg wit="f1823">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
			<rdg wit="fThomas">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
		</rdgGrp>
		<rdgGrp n="['shall', 'not', 'die!', '—', 'you,', 'my play-fellow, my companion,', 'my sister, perish on the scaffold! no! no!']">
			<rdg wit="f1831">shall not die! — You, &lt;longToken&gt;my play-fellow, my companion,&lt;/longToken&gt; &lt;longToken&gt;my sister, perish on the scaffold! No! no!&lt;/longToken&gt; </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['i']">
			<rdg wit="f1818">I </rdg>
			<rdg wit="f1823">I </rdg>
			<rdg wit="fThomas">I </rdg>
			<rdg wit="f1831">I </rdg>
			<rdg wit="fMS">I </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['never']">
			<rdg wit="fMS">never </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['never']">
			<rdg wit="f1818">never </rdg>
			<rdg wit="f1823">never </rdg>
			<rdg wit="fThomas">never </rdg>
			<rdg wit="f1831">never </rdg>
			<rdg wit="fMS">&lt;lb n=&quot;c56-0103__main__19&quot;/&gt;never </rdg>
		</rdgGrp>
	</app>

Without tokenizing hyphens and dashes: output example from C-14:

<app>
		<rdgGrp n="['must die –', 'you', 'my companion, my playfellow,', '', 'my more than sister—', 'die', '–i', 'never']">
			<rdg wit="fMS">&lt;longToken&gt;must die –&lt;/longToken&gt; &lt;lb n=&quot;c56-0103__main__17&quot;/&gt;you &lt;longToken&gt;my companion, my playfellow,&lt;/longToken&gt; &lt;lb n=&quot;c56-0103__main__18&quot;/&gt; &lt;longToken&gt;my more than sister—&lt;/longToken&gt; die –I never </rdg>
		</rdgGrp>
		<rdgGrp n="['must die;', 'you,', 'my', 'playfellow,', 'my', 'companion,', 'my more than sister.']">
			<rdg wit="f1818">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
			<rdg wit="f1823">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
			<rdg wit="fThomas">&lt;longToken&gt;must die;&lt;/longToken&gt; you, my playfellow, my companion, &lt;longToken&gt;my more than sister.&lt;/longToken&gt; </rdg>
		</rdgGrp>
		<rdgGrp n="['shall', 'not', 'die!—you,', 'my play-fellow, my companion,', 'my sister, perish on the scaffold! no! no!']">
			<rdg wit="f1831">shall not die!—You, &lt;longToken&gt;my play-fellow, my companion,&lt;/longToken&gt; &lt;longToken&gt;my sister, perish on the scaffold! No! no!&lt;/longToken&gt; </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['i']">
			<rdg wit="f1818">I </rdg>
			<rdg wit="f1823">I </rdg>
			<rdg wit="fThomas">I </rdg>
			<rdg wit="f1831">I </rdg>
		</rdgGrp>
	</app>
	<app>
		<rdgGrp n="['never']">
			<rdg wit="f1818">never </rdg>
			<rdg wit="f1823">never </rdg>
			<rdg wit="fThomas">never </rdg>
			<rdg wit="f1831">never </rdg>
			<rdg wit="fMS">&lt;lb n=&quot;c56-0103__main__19&quot;/&gt;never </rdg>
		</rdgGrp>
	</app>