[bug #58165] test-suite: "nroff -V ..." fails for other devices than "ascii"

[Bjarni Ingi Gislason]

URL:
<https://savannah.gnu.org/bugs/?58165>

Summary: test-suite: "nroff -V ..." fails for other devices
than "ascii"
Project: GNU troff
Submitted by: bjarniig
Submitted on: Sun 12 Apr 2020 08:37:06 PM UTC
Category: Core
Severity: 3 - Normal
Item Group: Test
Status: None
Privacy: Public
Assigned to: None
Open/Closed: Open
Discussion Lock: Any
Planned Release: None

_______________________________________________________

Details:

Subject: test-suite: "nroff -V ..." fails for other devices than "ascii"

All test cases in "src/roff/nroff/tests/verbose_option_works.sh" fail
with the "latin1" output device.

"nroff" can use the output devices ascii, cp1047, latin1, and utf8.

N.B.

Neither pre- nor post-processors are shown in the output of
"nroff -V ...".






_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[G. Branden Robinson]

Update of bug #58165 (project groff):

Status: None => Need Info
Assigned to: None => gbranden

_______________________________________________________

Follow-up Comment #1:


[comment #0 original submission:]

Post by Bjarni Ingi Gislason
Subject: test-suite: "nroff -V ..." fails for other devices than "ascii"
All test cases in "src/roff/nroff/tests/verbose_option_works.sh" fail
with the "latin1" output device.
"nroff" can use the output devices ascii, cp1047, latin1, and utf8.

I take it you are modifying the test case?

Post by Bjarni Ingi Gislason
Neither pre- nor post-processors are shown in the output of
"nroff -V ...".

nroff doesn't know what they are; only groff (the command) does.

_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[G. Branden Robinson]

Follow-up Comment #2, bug #58165 (project groff):

I took a guess at understanding your problem and cannot reproduce it. Locale
detection is working as documented, as far as I can tell.


$ PATH=.:$PATH ./nroff -V -Tascii
PATH=... groff -Tascii -mtty-char
$ PATH=.:$PATH ./nroff -V -Tlatin1
PATH=... groff -Tlatin1 -mtty-char
$ PATH=.:$PATH ./nroff -V -Tcp1047
PATH=... groff -Tcp1047 -mtty-char
$ PATH=.:$PATH ./nroff -V -Tutf8
PATH=... groff -Tutf8 -mtty-char
$ PATH=.:$PATH ./nroff -V
PATH=... groff -Tutf8 -mtty-char
$ LC_ALL=C PATH=.:$PATH ./nroff -V
PATH=... groff -Tascii -mtty-char
$ LC_ALL=C.UTF-8 PATH=.:$PATH ./nroff -V
PATH=... groff -Tutf8 -mtty-char


I need you to specify exactly what your test case was to be able to do
anything about this.

Please attach the output of the "locale" command on your failing test
environment, the body of the (presumably altered) test script, and the output
it produces.

_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Bjarni Ingi Gislason]

Follow-up Comment #3, bug #58165 (project groff):

Subject: test-suite: "nroff -V ..." fails for other devices than "ascii"

Sorry, I too often seem to neglect to provide relevant data.
I sometimes post findings before I test myself the matter further.
So you could wait for me providing more data when you ask for it.

1) First part of the test in the build directory:


# Ensure a predictable character encoding.
export LC_ALL=C
# Ensure a predictable command search path.

GROFF_RUNTIME="${GROFF_BIN_PATH=/usr/local/bin}:"

nroff="${abs_top_builddir:-.}/nroff"

echo "testing 'nroff -V'" >&2
expected="PATH=$GROFF_RUNTIME$PATH groff -Tascii -mtty-char"
actual=$("$nroff" -V)
diff -u <(echo "$expected") <(echo "$actual")

####

Result:

testing 'nroff -V'
--- /dev/fd/63 2020-04-13 01:11:32.936490725 +0000
+++ /dev/fd/62 2020-04-13 01:11:32.936490725 +0000
@@ -1 +1 @@
-PATH=/usr/local/bin:/home/bg/bin:/home/bg/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/sbin:/usr/sbin:/sbin
groff -Tascii -mtty-char
+PATH=/usr/local/bin:/home/bg/bin:/home/bg/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/sbin:/usr/sbin:/sbin
groff -Tlatin1 -mtty-char

####

Comment:

Added code to get the version of the nroff and the groff.
It was at first the version from Debian in /usr/bin/.

After changes, nroff was the latest in my build directory.

I do not install the software after each compilation.

My locale:

LANG=is_IS.iso88591
LANGUAGE=
LC_CTYPE="is_IS.iso88591"
LC_NUMERIC="is_IS.iso88591"
LC_TIME="is_IS.iso88591"
LC_COLLATE="is_IS.iso88591"
LC_MONETARY="is_IS.iso88591"
LC_MESSAGES="is_IS.iso88591"
LC_PAPER="is_IS.iso88591"
LC_NAME="is_IS.iso88591"
LC_ADDRESS="is_IS.iso88591"
LC_TELEPHONE="is_IS.iso88591"
LC_MEASUREMENT="is_IS.iso88591"
LC_IDENTIFICATION="is_IS.iso88591"
LC_ALL=


My charmap: ISO-8859-1


The make target "check" is not the same as the GNU standard (Debian
version is out of date) mentions:

'check'
Perform self-tests (if any). The user must build the program
before running the tests, but need not install the program; you
should write the self-tests so that they work when the program is
built but not installed.


There is also a "installcheck" named in the standard,
but it is empty in the Makefile of groff.


Suggestions:

Add the version of the used commands to the output of the tests.

Make "make check" work with the compiled versions in the build
directory.

N.B.

Neither pre- nor post-processors are shown in the output of
"nroff -V ...".

I expect that the same option in nroff and groff mean the same.


_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[G. Branden Robinson]

Follow-up Comment #4, bug #58165 (project groff):

I configured my system to compile the 8-bit Icelandic locale, and I simply
cannot reproduce this with the stock test.


$ locale
LANG=C
LANGUAGE=
LC_CTYPE="is_IS.ISO-8859-1"
LC_NUMERIC="is_IS.ISO-8859-1"
LC_TIME="is_IS.ISO-8859-1"
LC_COLLATE="is_IS.ISO-8859-1"
LC_MONETARY="is_IS.ISO-8859-1"
LC_MESSAGES="is_IS.ISO-8859-1"
LC_PAPER="is_IS.ISO-8859-1"
LC_NAME="is_IS.ISO-8859-1"
LC_ADDRESS="is_IS.ISO-8859-1"
LC_TELEPHONE="is_IS.ISO-8859-1"
LC_MEASUREMENT="is_IS.ISO-8859-1"
LC_IDENTIFICATION="is_IS.ISO-8859-1"
LC_ALL=is_IS.ISO-8859-1
$ bash src/roff/nroff/tests/verbose_option_works.sh
testing 'nroff -V'
testing 'nroff -V 1'
testing 'nroff -V "1a 1b"'
testing 'nroff -V "1a 1b" 2'
testing 'nroff -V 1a\"1b 2'
$ echo $?
0


I can get 'latin1' in the output of nroff -V if I try to run pieces of the
test script in isolation, and forget to set LC_ALL:


$ GROFF_RUNTIME="${GROFF_BIN_PATH=.}:" ./nroff -V
PATH=/usr/local/bin:/home/branden/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/games:/usr/games
groff -Tlatin1 -mtty-char
$ LC_ALL=C GROFF_RUNTIME="${GROFF_BIN_PATH=.}:" ./nroff -V
PATH=/usr/local/bin:/home/branden/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/games:/usr/games
groff -Tascii -mtty-char


In the above, my current working directory is the "build/" subdirectory of a
groff git checkout.

I suspect your problem is arising from extracting pieces of the test script
and failing to prepare the locale in conformance with its expectations.

I also strongly encourage you to migrate every Unix system under your control
to UTF-8.

_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[G. Branden Robinson]

Follow-up Comment #5, bug #58165 (project groff):

Here's some more stuff that works for me.


$ GROFF_RUNTIME="${GROFF_BIN_PATH=.}:" ./nroff -V
PATH=/usr/local/bin:/home/branden/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/games:/usr/games
groff -Tlatin1 -mtty-char
$ GROFF_RUNTIME="${GROFF_BIN_PATH=.}:" ./nroff /tmp/sister.ascii
A møøse once bið my sisþer...
$ GROFF_RUNTIME="${GROFF_BIN_PATH=.}:" ./nroff /tmp/sister.latin1
A møøse once bið my sisþer...
$ locale
LANG=C
LANGUAGE=
LC_CTYPE="is_IS.ISO-8859-1"
LC_NUMERIC="is_IS.ISO-8859-1"
LC_TIME="is_IS.ISO-8859-1"
LC_COLLATE="is_IS.ISO-8859-1"
LC_MONETARY="is_IS.ISO-8859-1"
LC_MESSAGES="is_IS.ISO-8859-1"
LC_PAPER="is_IS.ISO-8859-1"
LC_NAME="is_IS.ISO-8859-1"
LC_ADDRESS="is_IS.ISO-8859-1"
LC_TELEPHONE="is_IS.ISO-8859-1"
LC_MEASUREMENT="is_IS.ISO-8859-1"
LC_IDENTIFICATION="is_IS.ISO-8859-1"
LC_ALL=is_IS.ISO-8859-1


I'm attaching the input files as exhibits. One is ASCII-encoded and uses
groff character escapes; the other is encoded in ISO 8859-1.

(file #48826, file #48827)
_______________________________________________________

Additional Item Attachment:

File name: sister.ascii Size:0 KB
<https://savannah.gnu.org/file/sister.ascii?file_id=48826>

File name: sister.latin1 Size:0 KB
<https://savannah.gnu.org/file/sister.latin1?file_id=48827>



_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Bjarni Ingi Gislason]

Follow-up Comment #6, bug #58165 (project groff):


I looked at the issue again.

1) unset the LESSCHARSET environmental variable in the test script.

Result: correct operation.

2) looked at the first case test with "locale charmap" in "nroff".

Issued "LC_ALL=C locale charmap".

Result: ANSI_X3.4-1968

This shows that the real culprit is a missing case candidate in the
first case test, not the LESSCHARSET, nor the result of the "locale" I
use in the environment.

"groff" supports four output devices, so one is missing in the first
case test.

A case of wrong conclusions, suggestions is seen in the bug
report #58098.


_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[G. Branden Robinson]

Follow-up Comment #7, bug #58165 (project groff):


[comment #6 comment #6:]

Post by Bjarni Ingi Gislason
I looked at the issue again.
1) unset the LESSCHARSET environmental variable in the test script.

That doesn't sound like anything that's ever been part of the test script.
I'm the only one who's ever committed to it and I'm pretty sure.

It would be helpful if you'd try to reproduce problems against stock,
unmodified groffs when filing bugs. Telling us which git revision you're
testing wouldn't hurt either.

Post by Bjarni Ingi Gislason
Result: correct operation.

That's good to hear but a little puzzling. LESSCHARSET is a very distant
fallback. For it to be consulted, all locale checks have to fail.

Post by Bjarni Ingi Gislason
2) looked at the first case test with "locale charmap" in "nroff".
Issued "LC_ALL=C locale charmap".
Result: ANSI_X3.4-1968

Yes, that's what you should get.

Post by Bjarni Ingi Gislason
This shows that the real culprit is a missing case candidate in the
first case test, not the LESSCHARSET, nor the result of the "locale" I
use in the environment.
"groff" supports four output devices, so one is missing in the first
case test.

I don't follow any of your reasoning here.

1. When your locale is is_IS.ISO-8559-1, "locale charmap" will return
"ISO-8859-1" unless you override the command's environment, as in case 2
below.

2. When you run "LC_ALL=C locale charmap", you will always get back
ANSI_X3.4-1968".

3. You seem to have a system with a functioning locale system, so I don't see
why nroff would ever fall back to LESSCHARSET for you. Unless you're once
again doing something you haven't disclosed, like "LESSCHARSET=latin1 LC_ALL=C
nroff". In that scenario, nroff will attempt to use the latin1 device and it
is working exactly as documented in the man page.

I believe I've resolved the problems here with my latest commits to nroff and
its test script, but I'll give it a few minutes before resolving the ticket
because I need to catch up on mail.

Post by Bjarni Ingi Gislason
A case of wrong conclusions, suggestions is seen in the bug
report #58098.

_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

[Bjarni Ingi Gislason]

Follow-up Comment #8, bug #58165 (project groff):

1) My sentence can be interpreted in two ways,

more clearly:

I added the line

unset LESSCHARSET

to the script, as nroff checked for that case.

my environment has the variable defined with

LESSCHARSET=latin1

2) Add the line

export GROFF_TYPESETTER=ascii

to override a possible default value.

2) The test script contains the line

export LC_ALL=C

so a natural test is to check, how that affects the working of the
"nroff" script, in this case of "locale charmap".

The available locales on my computer are:

locale -a

C
C.UTF-8
icelandic
is_IS
is_IS.iso88591
is_IS.utf8
POSIX

I think the C locale is standard in GNU systems and thus always
available.

git version 2.25.1 (Debian testing)

The "nroff.sh" and "nroff.am" scripts have not been changed by me,
but I create a "test-nroff" from them with a "test-nroff.am" file, which
is based on the "nroff.am" file.



_______________________________________________________

Reply to this item at:

<https://savannah.gnu.org/bugs/?58165>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/