Issues with quail.el

[Amit Ramon]

I believe I found some bugs in quail.el that occurs when the keyboard
layout is set to a layout that is different from the standard
"quail-keyboard-layout-standard". I'd like to try and fix them myself,
and I have some questions.

First issue

Assume that the keyboard layout is set to "atari-german" and that the
input method is "Hebrew". When I type C-h I (thus invoking
"describe-input-method"), the drawing of the keyboard layout is
incorrect. The keys that appear in the shifted level, which should be,
in this input method, similar to those in the standard layout, are
actually taken from the current keyboard layout.

For example, the Hebrew letter TET is on the (standard) y key, and
pressing this key together with shift when Hebrew input method is
active should produce Y. This works fine when typing, but in the map
drawn by "describe-input-method" the letter that appears on this key
in the shifted state is Z, which is what this key does in
"atari-german" layout. There are some other keys not in the right
place - all those that in "atari-german" have a meaning that is
different than the one in the standard layout.

Second issue

The "quail-show-key" function should describe the keys need to be
pressed in order to produce the character at point. When the input
method is Hebrew and the keyboard layout is not the standard, this
functions fails to find the keys for a Hebrew letter that sits on a
key that its meaning in the current keyboard layout is different from
its meaning in the standard layout.

For example, if the keyboard layout is "atari-german", invoking
"quail-show-key" for the Hebrew letter TET produce the message "TET
can't be input by the current input method".

This is because the key on which TET sits is Y in the standard layout
but Z in "atari-german".

Questions and Comments

For the first issue, the problem is with the "quail-insert-kbd-layout"
function. In its first part this functions generate a list of
characters to be drawn, based in the input method. It walks along the
characters of the keyboard layout, try to translate each using
"quail-map" to obtain the input method character. If translation is
not found for a specific key, its value is taken from the *current
layout* given as an argument to the function. This, as far as I
understand, is an error - input methods are based on the standard
keyboard layout, or in other words, they are based on a fixed known
physical layout. Thus, if a certain input method omits specifying a
translation for a key, it expects that the "default" value will be
taken - i.e., the value from the standard layout.

I'd appreciate an advice here - if I'm right about this, it's an easy
fix.

I also have a more general question. This function takes an argument
which is a keyboard layout, and is used in quail exactly in two
places: one for drawing the layout of an input method (this is what I
was referring to so far), and the other is for drawing the layout of a
*keyboard layout*.

The first thing the function does is to translate the given layout to
the standard one, if "(quail-kbd-layout)" is t. I don't really
understand why this is needed - if input methods are based on the
standard layout, it seems to me that the "quail-insert-kbd-layout"
function don't need to receive a keyboard layout as an argument at
all. It should work directly with the standard layout and generate the
drawing from it (and that would solve the issue I raised).

So what is the situation in which the given keyboard layout should not
be translated back to the standard layout? If this is for drawing a
keyboard layout (not for an input method), then I would suggest to
break this function down to two separate functions.

I would appreciate your input about it.

For the second issue, I believe the problem is with the
"quail-find-key" function. This function takes a character and try to
back-translate it using quail's "decode-map". However, it seems the
"decode-map" is always based on the standard keyboard layout, no
matter what is the currently set layout. Thus, when the current layout
is not the standard, this function will fail for some characters.

I tried to fix it in a simple way - I take the key(s) returned from
the "decode-map", and translate them from the current layout to the
standard layout (a function for the other direction exists, I had to
write a new one). This seem to work.

However, as seen from the code, the decoding can return either a
string or a different type of object (cons maybe?). I'm not sure in
which cases a non-string type is returned and how to handle it.

With your help, I'd be happy to try and write a fix.

Thanks, and sorry for the length of this mail...

Amit

[Eli Zaretskii]

Date: Sat, 5 May 2018 14:53:57 +0300
Assume that the keyboard layout is set to "atari-german" and that the
input method is "Hebrew". When I type C-h I (thus invoking
"describe-input-method"), the drawing of the keyboard layout is
incorrect. The keys that appear in the shifted level, which should be,
in this input method, similar to those in the standard layout, are
actually taken from the current keyboard layout.

I'm no expert on this, but does the fact that Dvorak keyboards need
special input methods tell anything useful for your use case?

[Amit Ramon]

Hi,

Note, for example, the location of the "Y" and "Z", and the two keys
right of the "L". This layout is the standard for a Hebrew keyboard
layout (the Israeli standard, at least), and this is what a Hebrew
writer would expect when Hebrew input method is set. (you can get it
simply if you don't change your keyboard layout -- assuming it is the
standard one -- and then set input method to Hebrew, and do C-h I.)
The point is that *there are no problems with actual typing* -- you
press Shift with the key that has the "Z" in your drawing (to remove
any possible confusion, this is the key that is right below 6 and 7
and on the center between them), and you'll get "Y".

Strangely, though, that is not what I experience. Here are the
‎‪[phone].(\
‭׳קראזוןםפ][
‭שדגכעיחלך>=
‭טסבהנמצתץ&
QWERTZUIOP}{
YXCVBNM><_
There are a few that don't match, but the Y and Z are not two of them.
This is not the case for you?

This is what you actually get when typing, right? This is also the
case for me, but perhaps I didn't explain the problem clearly enough.

Just to make sure - I assume your physical keyboard is a standard
English (or at least very similar)? (i.e., it is not a real, physical,
atari-german keyboard.)

Also, before going on I suggest that we adopt some "naming
conventions" for keyboard keys, to make communication easier. I'll use
the convention described here:

https://en.wikipedia.org/wiki/ISO/IEC_9995#ISO/IEC_9995-1

It is similar to the way used for X Window keyboard maps. So, for
example, the row under the numbers is row D, and the keys are numbered
D1, D2 and so on from left to right, where "q" is on D1, "w" is on D2
and so on.

Lets first do a simple test. When Hebrew input method is active, you
press Shift-D6 and you get "Z" (per your result above). If you now
deactivate the input method ("toggle-input-method", bound by default
to C-\) and press D6 with and without Shift, do you get "y" and "Y"?

If so, then the "strangeness" is due to the fact that the physical
keyboard is not really atari-german -- if the keyboard was
atari-german, D6 in the test would print "z" and "Z".

Emacs receives just characters, it has no idea about the actual
physical keys that are pressed -- this is why we define for quail the
keyboard layout. It enables Emacs to "compute" the physical key that
was pressed by comparing the active keyboard layout with the standard
one.

We can "emulate" an atari-german real keyboard by pressing B1 when we
really mean D6, and vice versa. Then Emacs will receive "z" for
(emulated) "D6" and "y" for (emulated) "B1". Then we should swap the Z
and Y in your typing results.

I hope I managed to clear this out this time, although I'm not sure
how much I succeeded in this.

Best,

--- Amit

[K. Handa]

Very sorry for the late response.

When I try to recreate what you are showing, I get the following.
Does this match?
+----------------------------------------------------------------+
| 1! | 2" | 3§ | 4$ | 5% | 6& | 7/ | 8( | 9) | 0( | ß? | '` | ;^ |
+----------------------------------------------------------------+
| qQ | 'W | ק‎E | ר‎R | א‎T | ט‎Z | ו‎U | ן‎I | ם‎O | פ‎P | ]} | [{ |
+------------------------------------------------------------+
| ש‎A | ד‎S | ג‎D | כ‎F | ע‎G | י‎H | ח‎J | ל‎K | ך‎L | ף‎Ö | ,Ä | ~| |
+--------------------------------------------------------------+
| >< | ז‎Y | ס‎X | ב‎C | ה‎V | נ‎B | מ‎N | צ‎M | ת‎> | ץ‎< | ._ |
+------------------------------------------------------+

[...]

+----------------------------------------------------------------+
+----------------------------------------------------------------+
| /Q | 'W | ק‎E | ר‎R | א‎T | ט‎Y | ו‎U | ן‎I | ם‎O | פ‎P | ]} | [{ |
+------------------------------------------------------------+
| ש‎A | ד‎S | ג‎D | כ‎F | ע‎G | י‎H | ח‎J | ל‎K | ך‎L | ף‎: | ," | \| |
+-----------------------------------------------------------+
| ז‎Z | ס‎X | ב‎C | ה‎V | נ‎B | מ‎N | צ‎M | ת‎> | ץ‎< | .? |
+-------------------------------------------------+

[...]

Note, for example, the location of the "Y" and "Z", and the two keys
right of the "L". This layout is the standard for a Hebrew keyboard
layout (the Israeli standard, at least), and this is what a Hebrew
writer would expect when Hebrew input method is set. (you can get it
simply if you don't change your keyboard layout -- assuming it is the
standard one -- and then set input method to Hebrew, and do C-h I.)

It seems that the difference is at such a key that is different from the
standard layout and that is not mapped to Hebrew. So, I suspect that
the code (in quail-insert-kbd-layout) handling such a key should be
fixed. Could you please try the attached patch?

[...]

ט can't be input by the current input method

I confimed this bug, but it seems that fixing take more time. Please wait.

(add-to-list 'quail-keyboard-layout-alist
`("dvorak" . ,(concat " "
" '\",<.>pPyYfFgGcCrRlL/?=+ "
" aAoOeEuUiIdDhHtTnNsS-_\\| "
" ;:qQjJkKxXbBmMwWvVzZ "
" ")))
(quail-set-keyboard-layout "dvorak")
If you actually evaluate this elisp code, you could call
"quail-set-keyboard-layout" once with "dvorak" and once with
"standard" and compare the results of C-h I (and for that you don't
have to change the actual layout of the keyboard).
* Emacs has (see quail.el) definitions for sun-type3, atari-german,
pc102-de, jp106 and pc105-uk keyboard layouts only, but not Dvorak
(anyone has an idea why? perhaps it is time to add it?)

I agree.

[2] The definitions for the standard Hebrew input method ("hebrew" and
"hebrew-new") do not explicitly define the first shift level. Perhap
the author of it trusted quail to take this from the English standard
layout (the first shift level is identical, at least most of
it). However, perhaps it would be safer to completly define the Hebrew
layout, including the first shift level.

For that, I don't have a strong opinion. But, I think it is not good to
add extra code for hiding a bug of quail.

---
K. Handa
***@gnu.org

diff --git a/lisp/international/quail.el b/lisp/international/quail.el
index eece836354..a81244a1db 100644
--- a/lisp/international/quail.el
+++ b/lisp/international/quail.el
@@ -815,7 +815,7 @@ quail-insert-kbd-layout
(aref (cdr translation) 0)
" ")))
(setq done-list (cons translation done-list)))
- (setq translation (aref kbd-layout i)))
+ (setq translation ch))
(aset layout i translation))
(setq i (1+ i)))

[Amit Ramon]

Is there an input method for Southern China’s Cantonese for
- Jyutping w/Digits
- JP w/Superscripts
- Yale w/Marks
- Yale w/Numbers
phonetic system?
I have started to use chinese-tonepy and chinese-py-punct-b5 and the other possible completions to chinese- don’t look a match.

Hope I'm not saying the obvious, but

M-x list-input-methods

will display the list of all input methods.