terminal-unicode-core.tex

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{colortbl}
\usepackage{epigraph}
\usepackage{fancyhdr}
\usepackage{graphicx}
\usepackage{hhline}
\usepackage{todonotes}

\usepackage{csquotes}
\usepackage[backend=biber,style=alphabetic,sorting=anyt]{biblatex}
\addbibresource{refs.bib}

\usepackage[hidelinks]{hyperref}
\hypersetup{
	colorlinks=true,
	hypertexnames=true
	citecolor=red,
	linkcolor=blue,
	filecolor=magenta,
	citecolor=orange,
	urlcolor=cyan
}

\usepackage{draftwatermark}
\SetWatermarkText{Draft}
\SetWatermarkScale{4}

\usepackage{geometry}
\geometry{legalpaper, margin=1in}

\usepackage{xcolor}
\definecolor{light-gray}{gray}{0.95}

\title{Unicode in Terminals \\
a proposal to standardizing basic Unicode features}
\author{Christian Parpart}
\date{2021-09-06 (draft, revision 1)}

\newcommand{\code}[1]{\colorbox{light-gray}{\texttt{#1}}}

\newcommand{\Unicode}{\textbf{Unicode 13}}

\newcommand{\DECRQM}[1]{\code{CSI ? #1 \$ p} }
\newcommand{\DECSET}[1]{\code{CSI ? #1 h} }
\newcommand{\DECRST}[1]{\code{CSI ? #1 l} }

\newcommand\VtModeNum{2027}                	% mode Id that is used by this specification
\newcommand{\UCON}{\DECSET{\VtModeNum{}}}   % DECSM for enabling grapheme cluster processing
\newcommand{\UCOFF}{\DECRST{\VtModeNum{}}}  % DECRM for disabling grapheme cluster processing
\newcommand{\UCTEST}{\DECRQM{\VtModeNum{}}} % DECRQM for requesting current grapheme cluster processing mode

\begin{document}

\maketitle

\tableofcontents

\section{History and current state}

Historically, only 7-bit characters with C0 control codes
were supported by terminals and different languages
by selecting their respective code pages.

Later on this was extended to 8-bit ASCII and along with C1 control codes.

With the introduction of Unicode there were no need to have codepages anymore,
but the Unicode spec was not explicitly designed to also cover terminals,
except that C0 and C1 codepoints were preserved.

With Unicode UTF-8 it was possible to at least pass Unicode characters to the
terminal, but rendering of a few characters as well as their respective
cursor placement is not defined in the Unicode standard.

Also, Unicode introduced codepoint sequences that are mapping to
a single user perceived character - so called grapheme clusters.
The terminal has never attempted any formalization on how to deal with
grapheme clusters, variation selectors, their east asian width, nor
emoji and emoji presentation handling.

This spec tries to address some of the problems terminals are suffering
with Unicode today.

\section{Backwards Compatibility}

Backwards compatibility is retained by leaving everything as undefined
as it used to be without this specification when this mode is not enabled.

The application can test for the availability of this feature mode
and has to explicitly enable it in order to get the set of properties
as defined in this document guaranteed.

\section{Future Compatibility and Stability}

Unicode itself had a major breakage at version between version 8 and 9
with regards to some codepoints having their east asian width changed.

While this may happen any time again, we do not expect that to happen
that soon nor that frequent to address future incompatibilities
as of this spec and leave this for a later point.

\section{Feature and Mode State Detection}

\UCTEST \cite{DECRQM} can be used for testing the availability of this
feature as well as the current mode the terminal is in with regards
to this specification, the \UCTEST reply will indicate each state
accurately enough not to need any new VT sequence introduced.

\section{Mode Switching}

\begin{itemize}
	\item \UCON{} \cite{SM} for ensuring conformance to all rules as defined by this specification
	\item \UCOFF{} \cite{RM} for undefined behavior
\end{itemize}

\section{Semantics}

The following set of semantics \textbf{MUST} be adhered to if this
VT mode \code{\VtModeNum} is enabled.
If the VT mode \code{\VtModeNum} is not set, then the behavior is as undefined
as if this specification was not implemented at all in order to retain
behavior of current terminals and their legacy applications.

\subsection{Grapheme Cluster}

\paragraph*{}
With this mode enabled, the terminal \textbf{MUST} support grapheme clusters
in conformance to algorithm as described in \cite{UTS-29}.

\paragraph*{}
This implies that every consecutively written character on the terminal
stream that is non-breakable as per \cite{UTS-29} will
always end up in the same terminal's grid cell.

\paragraph*{}
Therefore, extending a grapheme cluster with consecutively added codepoints
will not move the cursor except for variation selector 16 (VS16) that may
have caused the width of the grapheme cluster to change to wide (2 grid cells).

\paragraph*{}
When the cursor moves to a grid cell that contains a complete or incomplete
grapheme cluster, this grid cell's contents will be erased and overwritten
rather then textually concatenated.

\paragraph*{}
Therefore cursor movement semantics of the terminal remain unchanged.

\subsection{Emoji}

\paragraph*{}
Emoji symbols are always rendered in square aspect ratio
(as proposed by \cite{UTS-51}),
implying a East Asian Width \cite{UTS-11} of Wide, 2 grid cells.

\paragraph*{}
ZWJ emoji are required to be displayed as a single image with a width of 2 grid cells.

The alternate display of ZWJ emoji in a decomposed sequence of sub-images
must not be used as a fallback as it will break cursor movement guarantees.

\paragraph*{}
If a ZWJ emoji cannot be rendered the display behavior is undefined -
for example, a Unicode replacement character \code{U+FFFD} could be
displayed instead.

\paragraph*{}
In emoji emoji presentation, the cursor will always move by 2 grid cells wide.

\paragraph*{}
SGR attributes applied to a grid cell containing an emoji symbol are
not strictly defined and it is left to the terminal emulator author to have
sensible and meaningful semantics with regards to emoji symbols.

\subsection{Variation Selector 16}

VS16 promotes the grapheme cluster to emoji emoji presentation,
implying that this will force the grapheme cluster's width to be 2,
which may possibly cause reflowing of that symbol to the next line
if on right margin with AutoWrap mode is set.

\subsection{Variation Selector 15}

\paragraph*{}
VS15 forces the grapheme cluster to emoji text presentation.
This will \textbf{NOT} change the underlying width
but only change the display to prefer textual non-colored presentation.

\paragraph*{}
This matches the behavior of todays web browsers and should thus
feel most intuitive to users.

\paragraph*{}
The cursor will move by columns if the symbol has the default presentation of emoji.

\subsection{Margins and AutoWrap with Emoji}

Emoji written at the right margin with \cite{DECAWM} AutoWrap mode disabled
may or may not be rendered in half or not be displayed at all.
This behavior is undefined to ease implementation and adoption 
of this specification.

\subsection{Terminal Reset}

Both, Soft reset \cite{DECSTR} as well as hard reset \cite{RIS}
will reset the mode back to undefined behavior.

\printbibliography[heading=bibintoc,title={References}]

\end{document}