Rendered at 04:42:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
joshdavham 2 days ago [-]
The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
WhyNotHugo 2 days ago [-]
I've seen Dutch comics for kids which use ÿ instead of ij. E.g.: "mÿn". In typical comic font (which is closer to handwriting), it looks similar to ij, but it's clearly ÿ.
pseudohadamard 2 days ago [-]
Great, so they added Ÿ but not n̈, making it very difficult to write music reviews.
(Not sure how the latter will render in your browser).
ComputerGuru 2 days ago [-]
iOS at least renders it as an n with dieresis; is that how it was intended (I’m unfamiliar with musical notation)? If so, what is so difficult about it?
In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
cryptonector 1 days ago [-]
> In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters [...]
Yes. But the font has to be designed to allow this.
That means that a) lower-case letters must be small enough to allow "overstrike" with diacritical marks to render correctly, b) diacritical marks must be small enough too, c) if you want capitals to also render correctly then the font must have either a set of smaller capitals, or smaller/higher diacritics, and the renderer must scale the capitals and diacritics to fit, or change line spacing, etc.
Also, the 'semantics' for the _human_ reader are the same whether you use pre-composed or decomposed codepoint sequences -- the semantics for the human are about the glyph as rendered and not the details of how that glyph was obtained.
And to be super-pedantic (sorry!), what you call 'modifiers' are called combining marks in Unicode, and what you call 'standalone characters' are called precompositions in Unicode. And it's not necessarily true that the rendering will _in practice_ work better with the former than the latter, but in theory absolutely it is, and in practice it almost always is for _browsers_.
> Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
I really wish Windows would adopt X11-style compose key sequences. Those are incredibly natural for all glyphs that can notionally be constructed via 'overstriking', and historically that is exactly how people did construct those with typewriters. (I don't know, but I suspect that for typesetting it was necessary to have a type for each modified character because having decomposed movable type would probably not have been robust enough.)
ComputerGuru 1 days ago [-]
Thanks for the corrections, but I am specifically talking about semantics from the Unicode Technical Committee's perspective, of the underlying Unicode codepoint(s). There is a reason some end-user-viewable glyphs can be formed in multiple ways, sometimes with standalone codepoints (precompositions, sure) and sometimes via the use of combining marks. You have to go back to the Unicode project's actual founding vision, and its basis for accepting new codepoints or declining to do so. People are surprised to learn it has little to do with what the human-visible end result looks like.
cryptonector 14 hours ago [-]
Can you fill us in on this history?
My understanding is that decompositions for Latin scripts was just natural typewriter-style constructions, while precompositions for Latin scripts was also natural to simplify table-driven transcoding between Unicode and ISO-8859.
Similar considerations probably applied in the case of Hiragana (I'm guessing) and other scripts.
Besides, combining marks (decomposition) allow for creating new glyphs based on existing ones even where Unicode does not define them.
Once two or more ways existed to write any given glyph the canonical equivalence problem immediately arose, and the UC was aware of it immediately, thus we get two basic NFs (NFD, NFC).
When it comes to the semantics of glyphs, whatever the UC intended is one thing, but how natural language evolves to use those glyphs is another. So to some degree what the UC intended is a footnote, and what matters is how people use Unicode.
n̈ renders fine on my browser, but there is no compose-key sequence for it on my system.
I suspect the reason it renders fine is that 'n' in the font I'm using is small enough that the combining mark can be rendered by "overstriking" a diaeresis.
pseudohadamard 1 days ago [-]
It seems to be font-related, when I edited the text it rendered fine but viewing the resulting page has the unlauts floating in space after the n.
cryptonector 14 hours ago [-]
I also see the same, and I suspect it is due to the lack of a precomposition for n̈.
cryptonector 1 days ago [-]
> but only as a y with a diæresis
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
mostlylurks 2 days ago [-]
> Ÿ seems to not be used by any language.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
NegativeLatency 2 days ago [-]
Not everything is upper/lower convertible like ß for example (as far as I know)
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
tengwar2 2 days ago [-]
Sure, but I've never heard of a diæresis being added to a capital letter.
seszett 2 days ago [-]
It is on signs when entering L'Haÿ-les-Roses at least. Every letter in French must have a capital version.
tengwar2 2 days ago [-]
Yup. Just checked on Streetview.
erjiang 2 days ago [-]
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
retrac 2 days ago [-]
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
thaumasiotes 2 days ago [-]
> I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin).
Huh? How do you pronounce 切?
> And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
matvore 2 days ago [-]
A component can contribute meaning as well as pronunciation. 沌 is pronounced like 屯 and they both have a kind of "gather" "clog" meaning
thaumasiotes 1 days ago [-]
That is not a method of character construction. It could happen, but you would never expect it.
Also... there is no "gather" or "clog" sense of 沌? There's a "gather" sense of 屯...
Where did you get the original 'example' of 沌 / 屯 from? Most people prefer to use examples that they have some kind of personal knowledge of.
> There are other meanings to it. [沌]
Not as many as you might expect. It's (part of) a mythological term referring to the state of the universe before it took the form we can observe today. It has a couple of other metaphorical meanings coming from that, or from traditional sayings related to that.
I see that wiktionary lists 沌がる as an alternate spelling of 塞がる "be blocked" [塞 - obstruct / block / plug], which looks like the closest thing we're likely to find to your earlier claim. Obvious problems using this to support your comment are:
- There's still no "gathering" or "accumulating" sense. (Nor is there an "obstruct" sense of 屯.)
- This is a meaning assigned long after the character came into existence, which means it cannot have informed the construction of the character.
matvore 1 days ago [-]
I learned about 沌 quite a while ago when I studied the first few chapters of Genesis in Japanese (混沌 was used to describe the earth in Gen 1:2) and something jogged my memory when I was looking for examples.
The dictionary link I posted had あつまる、水があつまる in the first definition. I don't know if that's a reliable dictionary. Maybe you think it's not?
> - This is a meaning assigned long after the character came into existence, which means it
> cannot have informed the construction of the character.
I don't know anything about which meanings were used first and which were added later, but what you say sounds believable.
> - There's still no "gathering" or "accumulating" sense. (Nor is there an "obstruct" sense of 屯.)
Let me clarify. Even without the first sense for 沌 in the above dictionary (あつまる、水があつまる) some of the other definitions for 沌 had "gather" included in it conceptually, namely clog and joined together without distinction.
I confirmed a few, and they seem correct. I thought 界 was the most easily understood example. 介 means being between two things and contributes the pronunciation. 界 means region, span, separate, division.
pezezin 2 days ago [-]
It is true that most hanzi are phonosemantic compounds; however, Japanese-created kanji are mostly semantic compounds. You can still guess the meaning, but good luck trying to guess the pronunciation.
The article mentions "an example of 彁 mistakenly used in a digitized Taisho newspaper due to a faded printing of 彊", but to me that implies the symbol already existed before then.
lidavidm 2 days ago [-]
Isn't that backwards? The nonexistent character is in the -digitized- version so presumably OCR or something got 彊 wrong, that's not saying that 彁 was used in the -print- version.
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
joemi 2 days ago [-]
I think you're both saying the same thing: the digitization of the article wouldn't have been the source, since 彁 would have had to exist before the digitization happened in order for the OCR to misread 彊 as 彁.
usrnm 2 days ago [-]
Most kanji are a combination of several smaller parts called "radicals" in English. If you look at these two kanji through this lens, you will see that it's actually a very simple mistake, one existing radical is replaced by another existing radical. It is very easy to imagine software that was working exactly like that: interpreting kanji as a combination of radicals rather than individual unrelated symbols
hunter2_ 2 days ago [-]
This reminds me of the pregnant man emoji [0] but it turns out they decided to handle that in an unusual way [1] so not really.
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
arcanemachiner 2 days ago [-]
That's not on my keyboard... Can I just stick with "ineffable"?
holdupagain 2 days ago [-]
Not exactly, it doesn't look like the definition or even connotations of "ineffable" include "unknown".
speerer 2 days ago [-]
It does in the sense that ineffable things are unable to be described in language (conceptually: unknown to language).
thaumasiotes 2 days ago [-]
Anything that is known can be described in language by giving it a name.
gweinberg 2 days ago [-]
Hey, that's today's smbc https://www.smbc-comics.com/comic/r
The problem is, there's no good reason to think the concept of 彊 in your mind is at all similar to the concept of 彊 in my brain. After all, we (probably) don't pronounce it the same, right?
thaumasiotes 1 days ago [-]
So learn what the word means. How did you learn what "concept" meant?
throw0101a 2 days ago [-]
> Anything that is known can be described in language by giving it a name.
And yet we have the phrase "It has a a certain je ne sais quoi."
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
CodesInChaos 2 days ago [-]
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
hnfong 2 days ago [-]
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
wodenokoto 2 days ago [-]
Han-unification was the result of Asian scholars, in particular the Ideaographic Research Group [1]
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Sorry, probably not very standard terminology. I somehow picked it up from the OReilly book: Fonts & Encodings: From Advanced Typography to Unicode and Everything in Between
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
podocarp 2 days ago [-]
Are there examples of characters with no glyphs and glyphs with multiple characters? Does it mean there is a motion of a certain word but nobody knows how to write it or there are multiple ways to write it? When going from traditional to simplified Chinese many words with different meanings were merged which is kinda weird. Like sprout 發 and hair 髮 now being a single word 发. Is that what its trying to say?
wodenokoto 2 days ago [-]
I haven’t been deep into Unicode for a long time, so I can’t remember all the intricacies of the word “glyph”.
So I’m not sure if there are glyphs with multiple characters but there _are_ code points with multiple characters. E.g, you can enter the correct code point and still misspell a Chinese or Japanese word. This is because for Chinese Korean and Japanese characters in Unicode it is not enough to choose the correct code point but also markup the code point with the correct language.
hnfong 2 days ago [-]
It's probably saying something like the space characters (eg. 0x20) have no glyphs. And examples of "glyphs with multiple characters" would be those characters that look exactly like the ASCII alphabets that lead to things like domain spoofing etc.
That's my interpretation, disclaimer I'm not an expert in this stuff.
bombcar 2 days ago [-]
I think the Aristotelian (though I might have reached for Platonic) idea is that there is an "A" which is essential, and that you map many various "A"s to that one A - which results in weirdness when you leave the Latin-alphabet based languages.
(Arguably even in English we run into the fact that there is not one "A" - there are many "A"s that sound and act completely different, and we collapsed them into one representation.)
riffraff 2 days ago [-]
Many times I have asked my Hungarian wife whether a sound is "a" or "a with an accent" (A vs Á), but they are different letters in Hungarian, not just variations of one as they are in Italian. We made a mess with "accent unification".
Which is always fun when you think that clearly nobody thinks of "o or o with a leg" (O vs Q).
saati 2 days ago [-]
That's not how Unicode works, A Α and А are different characters (latin, greek, cyrillic).
ButlerianJihad 2 days ago [-]
But for English speakers, there are many, many ways to interpret and pronounce that very Latin "A".
Many languages are strictly phonetic and their orthography has a very strict 1:1 ratio of glyph:sound. English is not one of those!
So in English, while we have five glyphs for the main vowels, those five glyphs have extremely versatile interpretations when actually converting into the spoken words.
throwaway2037 2 days ago [-]
Thank you for this post! Everytime I see another Online Armchair Ideograph Expert opine about the Han Unification effort as being "a mess", I roll my eyes. There was a large, well-respected team of linguists and cultural archeologists (I don't know the correct term to use here) that worked on that project. Whatever I will learn about Chinese characters in my life, I will never know as much as that group of scholars.
pamcake 2 days ago [-]
> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
yorwba 2 days ago [-]
Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
knuckleheadsmif 2 days ago [-]
I was involved in this before Unicode be a e Unicode when it started as discussions with Xerox and Apple—Xerox at that time had their XCCS standard. Those discussions directly morphed into Unicode. [[I was even there when we picked the name Unicode for what we had home to produce.]]
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Joker_vD 2 days ago [-]
Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what?
asveikau 2 days ago [-]
Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
Ah ok. So the Bulgarian printed lowercase looks similar to Russian cursive.
wl 2 days ago [-]
Unicode has done the same unification for cuneiform languages. If you’re dealing with texts in Hittite, Sumerian, or Akkadian, you need to make sure you’re using the right fonts, otherwise the text will look entirely wrong.
wodenokoto 2 days ago [-]
Same thing, different code points is the opposite of han unification. Han unification is same code point, different thing.
numpad0 2 days ago [-]
Unicode merges some Chinese and Japanese characters into same code points and leave others as they are, on random per-character basis. So Chinese guys gets a ton of Japanese mixed texts and it causes tons of problems.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.
numpad0 2 days ago [-]
I don't know but last I glanced at it, the Unification was implied to be coming from Chinese pressure than Japanese. Each of Kanji/Hanzi countries already had each own acceptable common use character sets and those could have been just populated into each own spaces. That seemed like the original plan at some point.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
throwaway2037 2 days ago [-]
> As far as I understood it, the result was an incoherent mess.
Do you have any specific examples? I never heard this before.
panzi 2 days ago [-]
Interesting. I guess there are no well working normalization functions as there are for the different code points that can make umlauts (äöü)?
charcircuit 2 days ago [-]
>having different code points so that things like text search sometimes don't work.
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
geokon 2 days ago [-]
Do you feel similarly about french and english? It could work but would be a little weird and annoyong to switch language modes
I think you underestimate the similarities. For instance Japanese names are not translated to Chinese. Theyre just read with Chinese pronunciations. So Chinese will regularly interact with Japanese content (im guess it happens the other way around too, but i dont have the personal experience)
charcircuit 2 days ago [-]
>Do you feel similarly about french and english?
Yes. If you search gâteau it shouldn't show results for cake by default.
>Japanese names are not translated to Chinese
Then it makes sense to search for the names in Japanese. Either they are translated to Chinese so you should be able to search with Chinese, or they aren't and you should search with Japanese.
geokon 2 days ago [-]
> Either they are translated to Chinese so you should be able to search with Chinese, or they aren't and you should search with Japanese.
The point is that when they look identical most of the time, you have no idea which language "mode" the text is in.
Tokyo is 東京 in both languages. They're not visually distinguishable. Maybe in a long list some particular characters are written slightly differently, but you'd have to really inspect the list and hope that distinguishing characters show up.
From the outside this may look weird, but to people that are around Chinese characters having multiple ways to write a character is just a normal fact of life. They look different in classical writing, seal scripts and cursive scripts. Trades people will also use shorthands. You have analogous situations with Simplified and Traditional Character - where some characters are simplified to fewer strokes and others are not. But as a reader you don't really care if it's 吃/喫 or 臺灣/台灣/台湾. There is basically no situation where you want to find 吃 but not 喫.
I get the desire to preserve native Japanese forms of the Chinese characters. But that seems like something mostly resolved with a font
If you want to mix Chinese and Japanese characters, then you are left having to mix fonts - which is a bit ugly I guess. It's not the most ergonomic solution, but this is the edge case. Most of the time you want different forms to search the same. If you search 臺灣 and your word processor skipped 台灣, then you'd be rightfully annoyed.
muvlon 2 days ago [-]
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
esrauch 2 days ago [-]
Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space.
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
Joker_vD 2 days ago [-]
We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.)
gucci-on-fleek 2 days ago [-]
Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings.
Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
Joker_vD 2 days ago [-]
Letterlike Symbols and Fullwidth Forms are in BMP (and I generously consider Enclosed Alphanumerics to not be a copy of a Latin alphabet).
gucci-on-fleek 2 days ago [-]
Letterlike Symbols is a bit of a weird grab bag of characters, most of which are clearly distinct from their "regular" Latin versions, but I do agree that a few of them are pointless exact copies of regular Latin letters.
The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :)
knuckleheadsmif 2 days ago [-]
That’s correct, it was deemed critical to making Unicode 16bits (although there were other good technical reasons but they would apply to unifying other languages as well.)
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
RicoElectrico 2 days ago [-]
The Han unification probably was about fitting CJK into the 16 bits aka BMP.
It was a blunder indeed.
knuckleheadsmif 2 days ago [-]
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
zx8080 2 days ago [-]
What is the rationale to the unifying direction of changes in Unicode?
sedatk 2 days ago [-]
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
eproxus 2 days ago [-]
This would make a great tattoo, as a pun on people getting Kanji character tattoos without being able to read Kanji.
xelxebar 2 days ago [-]
Xu Bing has a book that consists entirely of invented characters:
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
zahlman 2 days ago [-]
(2018)
I've seen YouTube videos on this topic before.
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.
It's from Unicode version 3.1 (published 2001) so this is surprising.
panzi 2 days ago [-]
Is anyone using these characters now for anything? No youth language or online slang using it?
decimalenough 2 days ago [-]
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
panzi 2 days ago [-]
It's something! I propose that the first 4 of those characters should in English be called Blinky, Pinky, Inky, and Clyde.
amake 2 days ago [-]
Should probably have "(2008)" in the title
Dwedit 2 days ago [-]
Saw headline, expected branch prediction vulnerability involving Unicode, was surprised at a completely different topic.
pocksuppet 2 days ago [-]
A language where you can make up new characters seems anathema to a universal character encoding.
podocarp 2 days ago [-]
So what is the spectre? I thought this was proposing something new like the manifesto
eviks 2 days ago [-]
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
zahlman 2 days ago [-]
> Why didn't they simly replace the original bad one?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
eviks 2 days ago [-]
But it's even more useful for people who want to use the actual character to be able to use it! And you can discuss using other means, not like drawings or old standard data disappears
Weird absolutism re error preservation instead of striving for correctness
zahlman 1 days ago [-]
> But it's even more useful for people who want to use the actual character to be able to use it!
... And they can, since a code point was assigned for it; so what's the problem?
eviks 1 days ago [-]
The problem is quoted right there in my original comment!
zahlman 20 hours ago [-]
There isn't any actual problem. People who want to use the character can still do so. If you're referring to
> was not added to JIS or Unicode until much later
then TFA is simply inaccurate; 𡚴 has been in Unicode since 2001. More importantly, replacing the incorrect character (instead of supplementing it) wouldn't realistically have made it possible to add it sooner. They didn't really know what they were doing back at the start.
gucci-on-fleek 2 days ago [-]
> Not that hard to imagine, OCR existed back then?
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
Izkata 2 days ago [-]
(Talking about Japanese here)
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
cyphar 2 days ago [-]
I find 鸚哥（いんこ） to be the easiest way to type the right hand side of 彁.
eviks 2 days ago [-]
It does exist? It's part of Unicode!
gucci-on-fleek 2 days ago [-]
It's listed in the Unicode character database, but I doubt that it was in (m)any fonts 20 years ago, it would have been in zero printed books [0], and nobody knew the meaning of the character, so I'd argue that it's more an artifact of the Unicode compilation process than a "real" character.
[0]: Aside from the "Overview of National Administrative Districts" book mentioned in the article
eviks 2 days ago [-]
You don't need (m)any fonts, only need one font used by the book and you can get it the same way it was originally done - gluing some parts of other characters together. Or you could just draw it if the Unicode version is too dissimilar from the book version for a reliable OCR.
(though if the character came from this single book, likely the Unicode reference is its replica?)
Kye 2 days ago [-]
OCR was slow and unreliable and was for a very long time.
eviks 2 days ago [-]
"slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation!
Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
ComputerGuru 2 days ago [-]
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way.
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
philipov 2 days ago [-]
"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."
schoen 2 days ago [-]
Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
somat 2 days ago [-]
I have it it in my .XCompose as
<Multi_key> <c> <c> <c> <p> : "\xe2\x98\xad" #symbol representing proletarian solidarity between agricultural and industrial workers
Never used, but I laugh every time I see it there.
As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
(Not sure how the latter will render in your browser).
In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
Yes. But the font has to be designed to allow this.
That means that a) lower-case letters must be small enough to allow "overstrike" with diacritical marks to render correctly, b) diacritical marks must be small enough too, c) if you want capitals to also render correctly then the font must have either a set of smaller capitals, or smaller/higher diacritics, and the renderer must scale the capitals and diacritics to fit, or change line spacing, etc.
Also, the 'semantics' for the _human_ reader are the same whether you use pre-composed or decomposed codepoint sequences -- the semantics for the human are about the glyph as rendered and not the details of how that glyph was obtained.
And to be super-pedantic (sorry!), what you call 'modifiers' are called combining marks in Unicode, and what you call 'standalone characters' are called precompositions in Unicode. And it's not necessarily true that the rendering will _in practice_ work better with the former than the latter, but in theory absolutely it is, and in practice it almost always is for _browsers_.
> Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
I really wish Windows would adopt X11-style compose key sequences. Those are incredibly natural for all glyphs that can notionally be constructed via 'overstriking', and historically that is exactly how people did construct those with typewriters. (I don't know, but I suspect that for typesetting it was necessary to have a type for each modified character because having decomposed movable type would probably not have been robust enough.)
My understanding is that decompositions for Latin scripts was just natural typewriter-style constructions, while precompositions for Latin scripts was also natural to simplify table-driven transcoding between Unicode and ISO-8859.
Similar considerations probably applied in the case of Hiragana (I'm guessing) and other scripts.
Besides, combining marks (decomposition) allow for creating new glyphs based on existing ones even where Unicode does not define them.
Once two or more ways existed to write any given glyph the canonical equivalence problem immediately arose, and the UC was aware of it immediately, thus we get two basic NFs (NFD, NFC).
When it comes to the semantics of glyphs, whatever the UC intended is one thing, but how natural language evolves to use those glyphs is another. So to some degree what the UC intended is a footnote, and what matters is how people use Unicode.
I suspect the reason it renders fine is that 'n' in the font I'm using is small enough that the combining mark can be rendered by "overstriking" a diaeresis.
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
Huh? How do you pronounce 切?
> And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
Also... there is no "gather" or "clog" sense of 沌? There's a "gather" sense of 屯...
There are other meanings to it.
I admit I had a hard time finding an example.
Where did you get the original 'example' of 沌 / 屯 from? Most people prefer to use examples that they have some kind of personal knowledge of.
> There are other meanings to it. [沌]
Not as many as you might expect. It's (part of) a mythological term referring to the state of the universe before it took the form we can observe today. It has a couple of other metaphorical meanings coming from that, or from traditional sayings related to that.
I see that wiktionary lists 沌がる as an alternate spelling of 塞がる "be blocked" [塞 - obstruct / block / plug], which looks like the closest thing we're likely to find to your earlier claim. Obvious problems using this to support your comment are:
- There's still no "gathering" or "accumulating" sense. (Nor is there an "obstruct" sense of 屯.)
- This is a meaning assigned long after the character came into existence, which means it cannot have informed the construction of the character.
The dictionary link I posted had あつまる、水があつまる in the first definition. I don't know if that's a reliable dictionary. Maybe you think it's not?
> - This is a meaning assigned long after the character came into existence, which means it > cannot have informed the construction of the character.
I don't know anything about which meanings were used first and which were added later, but what you say sounds believable.
> - There's still no "gathering" or "accumulating" sense. (Nor is there an "obstruct" sense of 屯.)
Let me clarify. Even without the first sense for 沌 in the above dictionary (あつまる、水があつまる) some of the other definitions for 沌 had "gather" included in it conceptually, namely clog and joined together without distinction.
This phenomenon we are talking about is called 会意兼形声文字 and this site seems to have a lot of accurate examples: https://okjiten.jp/20-kaiikenkeiseimoji.html#google_vignette
I confirmed a few, and they seem correct. I thought 界 was the most easily understood example. 介 means being between two things and contributes the pronunciation. 界 means region, span, separate, division.
https://en.wikipedia.org/wiki/Kokuji
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
[0] https://emojipedia.org/pregnant-man
[1] https://www.reddit.com/r/technology/comments/u7x3l9/comment/...
And yet we have the phrase "It has a a certain je ne sais quoi."
* https://lingoculture.com/blog/culture/je-ne-sais-quoi/
You can recognize/know that something is special, but cannot quite put your finger on why.
Perhaps the meaning of this character is sealed in some vault
Major previous discussions:
110 comments: https://news.ycombinator.com/item?id=17637375
130 comments: https://news.ycombinator.com/item?id=24951130
180 comments: https://news.ycombinator.com/item?id=32095502
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Seems to contradict each other.
[1] https://en.wikipedia.org/wiki/Ideographic_Research_Group
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
So I’m not sure if there are glyphs with multiple characters but there _are_ code points with multiple characters. E.g, you can enter the correct code point and still misspell a Chinese or Japanese word. This is because for Chinese Korean and Japanese characters in Unicode it is not enough to choose the correct code point but also markup the code point with the correct language.
That's my interpretation, disclaimer I'm not an expert in this stuff.
(Arguably even in English we run into the fact that there is not one "A" - there are many "A"s that sound and act completely different, and we collapsed them into one representation.)
Which is always fun when you think that clearly nobody thinks of "o or o with a leg" (O vs Q).
Many languages are strictly phonetic and their orthography has a very strict 1:1 ratio of glyph:sound. English is not one of those!
So in English, while we have five glyphs for the main vowels, those five glyphs have extremely versatile interpretations when actually converting into the spoken words.
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
I think you underestimate the similarities. For instance Japanese names are not translated to Chinese. Theyre just read with Chinese pronunciations. So Chinese will regularly interact with Japanese content (im guess it happens the other way around too, but i dont have the personal experience)
Yes. If you search gâteau it shouldn't show results for cake by default.
>Japanese names are not translated to Chinese
Then it makes sense to search for the names in Japanese. Either they are translated to Chinese so you should be able to search with Chinese, or they aren't and you should search with Japanese.
The point is that when they look identical most of the time, you have no idea which language "mode" the text is in.
Tokyo is 東京 in both languages. They're not visually distinguishable. Maybe in a long list some particular characters are written slightly differently, but you'd have to really inspect the list and hope that distinguishing characters show up.
From the outside this may look weird, but to people that are around Chinese characters having multiple ways to write a character is just a normal fact of life. They look different in classical writing, seal scripts and cursive scripts. Trades people will also use shorthands. You have analogous situations with Simplified and Traditional Character - where some characters are simplified to fewer strokes and others are not. But as a reader you don't really care if it's 吃/喫 or 臺灣/台灣/台湾. There is basically no situation where you want to find 吃 but not 喫.
I get the desire to preserve native Japanese forms of the Chinese characters. But that seems like something mostly resolved with a font
If you want to mix Chinese and Japanese characters, then you are left having to mix fonts - which is a bit ugly I guess. It's not the most ergonomic solution, but this is the edge case. Most of the time you want different forms to search the same. If you search 臺灣 and your word processor skipped 台灣, then you'd be rightfully annoyed.
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :)
And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
https://en.wikipedia.org/wiki/A_Book_from_the_Sky
I've seen YouTube videos on this topic before.
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.
It's from Unicode version 3.1 (published 2001) so this is surprising.
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
... And they can, since a code point was assigned for it; so what's the problem?
> was not added to JIS or Unicode until much later
then TFA is simply inaccurate; 𡚴 has been in Unicode since 2001. More importantly, replacing the incorrect character (instead of supplementing it) wouldn't realistically have made it possible to add it sooner. They didn't really know what they were doing back at the start.
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
[0]: Aside from the "Overview of National Administrative Districts" book mentioned in the article
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
never mind i pressed caps lock everything is better now