The experience of rendering Arabic typography and its technical debt

The complexities of mixed LR and RL text are quite astonishing since it’s not really even a case of just switching modes when switching scripts since double-nested (or more) texts can change the semantics of line breaks. This article provides a good overview: https://tug.org/TUGboat/tb08-1/tb17knutmix.pdf [1]

In college [2], when I wanted to quote some texts from Exodus in Hebrew in a paper that I wrote, I ended up avoiding the issue by hand-reversing the letter order and manually breaking lines. 8 bits is insufficient to cover all the possible combinations of letters and vowel markings so the font didn’t include any vowel markings and only did dageshim for בּ and פּ if I recall correctly.

⸻

1. As an aside, it would have been really nice if Unicode provided a R-L mirrored Latin alphabet to make it easier for monolingual developers to grasp the complexities surrounding mixed directional typesetting. I suppose it could still be added, although Unicode tends towards conservatism on adding additional characters.

2. This was 1990, well before Unicode in the era of a hundred or so 8-bit character encodings, most of which were not implemented widely. I also had to type the text using the arbitrary ASCII-Hebrew mapping of the font I was using which, among other things, led me to discover that letter frequency in Hebrew is much more uniform than it is in English.

I was lucky enough to date someone from the Gulf, so I forced myself to understand and read Arabic scripts (but not understand the text), which is a huge bonus when creating multi-lingual designs.

I already had a good understand of CJK scripts, and you'll come across RtL there, with things like tategaki which is both vertical and RtL at the same time (and can include quotes in other languages such as English and Arabic). Here's some lyrics I made in that format for reference:

https://codepen.io/kingcharlesone/pen/GgRXLoM

What peculiarities does Cyrillic text have? I've never learned to convert Cyrillic to Latin.

I recently learned how messed up URLs are in RtL and I'm also eternally grateful to not have to deal with that. Simulated example:

    https://example.com/[Arabic or Hebrew start of sentence]
                                            wrapped_url_part

It seems like there are probably some phishing attacks based on this.

Monday's editor does this even for English text!

Both Latin and Chinese have been modified by the technology used to write them.

When carved in stone the lines are much straighter. When written with brush or pen they became semi-cursive. When printing was introduced, they became grid-like and regular.

What westerners who are passingly familiar would think of as the standard Chinese typeface - the strict square grid with straight-line characters - arises in part from printing technology. Easy to carve that into wood blocks, and easy to line up the slots into a grid.

Latin was similarly morphed to fit into the realities of printing in the 1500s. And is still being morphed. Notice how numbers 123... are in-line and at the same height as the letters. That's a very modern convention, typewriter and computer influence on our orthography. Traditionally digits were more likely to appear as subscript, off-centre.

Conversely, English has a joined form(cursive) that is nearly dead because mechanical text assistance devices (first typewriters, now computers) work much better with the block form. While sad in a cultural loss sort of way the joined form only really makes sense when the text is hand written.

I am not familiar with the history of Arabic typography, but I sort of assume there was an archaic block form and their current joined form is the result of many centuries of encoding hand writing practice. advanced enough that falling back to a block form is impossible with the side effect of making simple mechanical text formatting also impossible.

As for Chinese derived characters. we currently are able to jam them awkwardly into our alphabet optimized structures(one code per character) but I wonder if a Chinese native encoding would look different. Would it make sense to try and represent the sub-characters present in each Chinese character in the encoding? I suspect not, Chinese works, but it also does not appear amiable to simple mechanical assistance.

CJK languages can include vertical and RtL stretches too, to complicate matters. Here's some lyrics I made as a test:

https://codepen.io/kingcharlesone/pen/GgRXLoM

Japanese magazines usually mix three different script types on a majority of the pages like this:

https://imgur.com/a/x61XbIV

(In another quirk some Japanese mags open right-bound, others open left-bound)

awesome article! i appreciated the alternatives and misc. workarounds to OT/unicode typesetting toward the end, very helpful :D (go blue!)

It does seem like a KP-like algorithm ought to be able to optimize the break positions without extreme algorithmic difficulty aside from the inputs being considerably more complex than for Latin block print: the cost function for a proposed line is a straightforward [0] calculable function of the contents of the line, and I think one could make a dynamic programming algorithm that tracks, for each input position, the cost of the optimal layout of all text up to that position with a break at that position. This gives an algorithm that takes cubic time. (For input length n, you need to fill in n values in the table. Each value scans the entire table before that position and does a calculation with complexity linear in the proposed line length.)

As a practical matter, there’s an input length n and there is some upper bound B on a credible line length as measured in code points, so there are only at most n*B credible proposed lines to evaluate, which also limits the useful look back on the table to B positions, so I think the time complexity could be reduced to O(n*B^2) without making the results worse on reasonable inputs, and this is probably quite tolerable.

[0] Straightforward once you’ve implemented the whole Arabic rendering stack, anyway. I am certainly not qualified to calculate this function :)

> but when rendering you could stretch individual glyphs to the desired width

“Individual glyphs” :)

It’s Arabic, so you wouldn’t stretch a single glyph, id would have to e done after shaping so you can work out the next run (either a single Aleph or the joined characters) in order to know what is stretchable (then throw it to your layout step)

Learning language is like learning poetry. Your appreciation of the thing grows proportionally with the practice. Which means you don't need to learn to fluency to appreciate Arabic. Even modest practice, just learning to write the alphabet, will increase your appreciation for the language if that's what you'd like to do.

It's funny -- I hate the way Arabic sounds when spoken, even though I agree it looks pretty. To my ears French and Japanese sound the nicest.

This was once attempted, and in a way such that each letter needed only one character no matter it's position in the word

Unfortunately it died

https://worksthatwork.com/6/unified-arabic

maybe AI will solve this ?

> Also, it's depressing how hundreds of millions of people couldn't even get their language typeset on a computer, and our industry meanwhile was busy building AI-native AI for your groceries (have we mentioned it has AI btw?) and similar performative bullshit.

AI also brought you this "wonderful" article, I would note.

This is a thing I have wondered about. are arabic numbers little endian when embeded in arabic text? The text is read right to left. But the numbers are the same as we put them in our left to right text where they are big-endian(read big to small). so are numbers in original arabic little-endian? read small to large. with the interesting side effect that numbers are universal.

I think he's talking about the rendering algorithm with regards to the stream of text. essentially saying rendering direction should follow reading convention.

on the other hand, in formal arabic, it's not unusual that numers are read in clusters from least significant to most significant (right to left). 1984 would be read : eighty four and nine hundred and a thousand. not sure if the author is aware of this

Computers are tools that exist to serve humans. Humans should never be asked to become subservient to computers. Your suggestion is ludicrous and risible.

That might be because of translations from Arabic. The article was also posted on a different website where the author responded

> the Kashida section was contributed to this post from a talk in Arabic of Nawal Hadeed, which she translated and added to the post herself. Although I'm unsure of LLM usage in the translation process, looking at the original Arabic I felt some change in tone while editing the post. I could have either declined the translation and never have this documented, procrastinate in translating it myself (which has been ongoing for a while), or publish as it is. I found the last least damaging.

Same, and it is a shame. Reading it is wearisome.

But a message about your Door dasher also likely wouldn't be justified.