Charset problems with mutt 1.5.18

Before I submit a bug report, I would like to know if I'm missing
something obvious. Capital umlauts (ÄÖÜ) and sz (ß) are displayed
incorrectly here with mut 1.5.18+nntp+compressed on my Mac OS X
10.5.2 box. They look fine with 1.5.17, the same .muttrc, the same
system. Here's more information, if you need more details, please
tell me so:

$ ./mutt -v
Mutt 1.5.18 (2008-05-17)
Copyright (C) 1996-2008 Michael R. Elkins and others.
Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
Mutt is free software, and you are welcome to redistribute it
under certain conditions; type `mutt -vv' for details.

System: Darwin 9.2.2 (i386)
ncurses: ncurses 5.6.20061217 (compiled with 5.6)
libiconv: 1.11
hcache backend: GDBM version 1.8.3. 10/15/2002 (built Apr 2 2008 00:37:04)
Compile options:
-DOMAIN
-DEBUG
-HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE
+USE_FCNTL -USE_FLOCK
+USE_POP +USE_NNTP +USE_IMAP +USE_SMTP -USE_GSS +USE_SSL_OPENSSL -USE_SSL_GNUTLS -USE_SASL +HAVE_GETADDRINFO
+HAVE_REGCOMP -USE_GNU_REGEX
+HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET
+HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM
+CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME
-EXACT_ADDRESS -SUN_ATTACHMENT
+ENABLE_NLS -LOCALES_HACK +COMPRESSED +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR
+HAVE_ICONV -ICONV_NONTRANS -HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE
-ISPELL
SENDMAIL="/usr/sbin/sendmail"
MAILPATH="/var/mail"
PKGDATADIR="/usr/local/share/mutt"
SYSCONFDIR="/usr/local/etc"
EXECSHELL="/bin/sh"
-MIXMASTER
To contact the developers, please mail to <mutt-dev@mutt.org>.
To report a bug, please visit http://bugs.mutt.org/.

patch-1.5.18.rr.compressed.1
vvv.nntp


-André

--
May as well be hung for a sheep as a lamb!
Linkstation/KuroBox/HG/HS/Tera Kernel 2.6/PPC from <http://hvkls.dyndns.org>

Hi.

On Wed, 21 May 2008, André Berger <andre.berger@web.de> wrote:

> Capital umlauts (ÄÖÜ) and sz (ß) are displayed
> incorrectly here with mut 1.5.18+nntp+compressed on my Mac OS X
> 10.5.2 box. They look fine with 1.5.17, the same .muttrc, the same
> system. Here's more information, if you need more details, please
> tell me so:
>
> $ ./mutt -v
> Mutt 1.5.18 (2008-05-17)

> System: Darwin 9.2.2 (i386)
> ncurses: ncurses 5.6.20061217 (compiled with 5.6)
> libiconv: 1.11
> hcache backend: GDBM version 1.8.3. 10/15/2002 (built Apr 2 2008 00:37:04)
> Compile options:
> -DOMAIN
> -DEBUG
> -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE
> +USE_FCNTL -USE_FLOCK
> +USE_POP +USE_NNTP +USE_IMAP +USE_SMTP -USE_GSS +USE_SSL_OPENSSL -USE_SSL_GNUTLS -USE_SASL +HAVE_GETADDRINFO
> +HAVE_REGCOMP -USE_GNU_REGEX
> +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET
> +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM
> +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME
> -EXACT_ADDRESS -SUN_ATTACHMENT
> +ENABLE_NLS -LOCALES_HACK +COMPRESSED +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR
> +HAVE_ICONV -ICONV_NONTRANS -HAVE_LIBIDN +HAVE_GETSID +USE_HCACHE
> -ISPELL
> -MIXMASTER
> To contact the developers, please mail to <mutt-dev@mutt.org>.
> To report a bug, please visit http://bugs.mutt.org/.
>
> patch-1.5.18.rr.compressed.1
> vvv.nntp

If I remember correctly I had a similar problem on my solaris mutt some
versions ago. Although it did not look very plausible to me it helped to
change the +HAVE_WC_FUNCS to -HAVE_WC_FUNCS. Do this either by moving
away the cursesw libs during configure run or edit the config.h after
running configure.

You could also try to compile against slang instead of curses.

For me everything works fine with this configuration:

#v+
Mutt 1.5.18 (2008-05-17)
Copyright (C) 1996-2008 Michael R. Elkins and others.
Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
Mutt is free software, and you are welcome to redistribute it
under certain conditions; type `mutt -vv' for details.

System: SunOS 5.9 (sun4u)
slang: 20103
libiconv: 1.11
Compile options:
-DOMAIN
-DEBUG
-HOMESPOOL -USE_SETGID -USE_DOTLOCK -DL_STANDALONE
-USE_FCNTL -USE_FLOCK
-USE_POP +USE_IMAP +USE_SMTP -USE_GSS +USE_SSL_OPENSSL -USE_SSL_GNUTLS -USE_SASL +HAVE_GETADDRINFO
+HAVE_REGCOMP -USE_GNU_REGEX
+HAVE_COLOR -HAVE_START_COLOR -HAVE_TYPEAHEAD -HAVE_BKGDSET
-HAVE_CURS_SET -HAVE_META -HAVE_RESIZETERM
+CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME -CRYPT_BACKEND_GPGME
-EXACT_ADDRESS -SUN_ATTACHMENT
-ENABLE_NLS -LOCALES_HACK +COMPRESSED -HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR
+HAVE_ICONV -ICONV_NONTRANS -HAVE_LIBIDN +HAVE_GETSID -USE_HCACHE
-MIXMASTER
To contact the developers, please mail to <mutt-dev@mutt.org>.
To report a bug, please visit http://bugs.mutt.org/.

patch-1.5.18.rr.compressed.1
patch-1.5.13.tamo.w3mface.2
vvv.initials
vvv.quote
#v-

Cheers,

Thomas

--
slrn packages for ubuntu: http://www.foory.de/thw/slrn/

* Cameron Simpson (2008-05-29):
> [ Apologies to comp.mail.mutt - I know my References: header is
> busted, and will fix it sometime. - Cameron
> ]
>
> On 21May2008 16:38, André Berger <andre.berger@web.de> wrote:
> | Before I submit a bug report, I would like to know if I'm missing
> | something obvious. Capital umlauts (ÄÖÜ) and sz (ß) are displayed
> | incorrectly here with mut 1.5.18+nntp+compressed on my Mac OS X
> | 10.5.2 box. They look fine with 1.5.17, the same .muttrc, the same
> | system. Here's more information, if you need more details, please
> | tell me so:
>
> I think we may. I have just viewed this message on my Mac running Mac OS X
> 10.5.2 using mutt-1.5.18, using both the standard Mac Terminal program and
> the iTerm terminal emulator. I lack the nntp and compressed patches, but have
> trouble imagining them as the cause.
>
> Your capital umlauts and sz view just fine. I'll append my "mutt -v"
> below for comparison.

I've asked the question on mutt-devel in the meantime and have been
able to narrow the problem down to "--enable-gpgme". Clean sources
with

./configure --enable-gpgme && make all && ./mutt -n -F /dev/null -f ~/Mail/mbox

are already problematic. With --enable-gpgme left out, it works, also
with all my patches and configuration options applied.

I've got MacPorts 1.6.0/up'd. installed,

$ port info gpgme
gpgme 1.1.4,

whose mutt 1.5.17 variants all work OK for me as well.

> Can you describe _how_ the glyphs are misrendered? I expect there's some
> kind of locale or presumed charset issue. Do you have charset settings
> in your mutt config?

The uppercase Umlauts (ÄÖÜ) and sz (ß) are displayed incorrectly in
the index and the pager, as "?~D", "?~V", "?~\"; "?~_", even those I
have sent myself from mutt. The lowercase Umlauts are OK.

I somehow lost Douglas'(?) post here but did as he advised, manually
changing

#define HAVE_NCURSES_H 1

to

#define HAVE_NCURSESW_NCURSES_H 1

with no effect. Haven't tried slang yet (partly because I don't like
the looks of it), but I will later.

-André

On Thu, 29 May 2008, André Berger <andre.berger@web.de> wrote:

>> Can you describe _how_ the glyphs are misrendered? I expect there's some
>> kind of locale or presumed charset issue. Do you have charset settings
>> in your mutt config?

> The uppercase Umlauts (ÄÖÜ) and sz (ß) are displayed incorrectly in
> the index and the pager, as "?~D", "?~V", "?~\"; "?~_", even those I
> have sent myself from mutt. The lowercase Umlauts are OK.

This looks like the is a mismatch between locale settings and the
terminal and/or mutt.

What does 'locale' procude? What says ":set display" from within the
mutt commandline.

> I somehow lost Douglas'(?) post here but did as he advised, manually
> changing
>
> #define HAVE_NCURSES_H 1
>
> to
>
> #define HAVE_NCURSESW_NCURSES_H 1
>
> with no effect.

No. It should be the other way round. You already have
#define HAVE_NCURSESW_NCURSES_H 1
in your config.h, otherwise there wouldn't be a +HAVE_WC_FUNCS.

> Haven't tried slang yet (partly because I don't like
> the looks of it), but I will later.

I can't see the slightest difference in mutt between a version with
slang or a version with ncurses.

Cheers,

Thomas

--
[X] Nail here for new Monitor

* Thomas Wiegner (2008-05-29):
> On Thu, 29 May 2008, André Berger <andre.berger@web.de> wrote:

Thomas, I'm very sorry not to have been able to recall your name
correctly

> >> Can you describe _how_ the glyphs are misrendered? I expect there's some
> >> kind of locale or presumed charset issue. Do you have charset settings
> >> in your mutt config?
>
> > The uppercase Umlauts (ÄÖÜ) and sz (ß) are displayed incorrectly in
> > the index and the pager, as "?~D", "?~V", "?~\"; "?~_", even those I
> > have sent myself from mutt. The lowercase Umlauts are OK.
>
> This looks like the is a mismatch between locale settings and the
> terminal and/or mutt.

The thing is, everything is fine without --enable-gpg-me.

The difference between the config.h files:

< #define CRYPT_BACKEND_GPGME 1
> /* #undef CRYPT_BACKEND_GPGME */
< #define HAVE_GPGME_PKA_TRUST 1
> /* #undef HAVE_GPGME_PKA_TRUST */

> What does 'locale' procude?

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
$ set | grep -i term
TERM=xterm-color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=240

> What says ":set display" from within the
> mutt commandline.

display: unknown variable
charset="utf-8"
assumed_charset="windows-1252:x-mac-roman:utf-8"

> > I somehow lost Douglas'(?) post here but did as he advised, manually
> > changing
> > #define HAVE_NCURSES_H 1
> > to
> > #define HAVE_NCURSESW_NCURSES_H 1
> > with no effect.
>
> No. It should be the other way round. You already have
> #define HAVE_NCURSESW_NCURSES_H 1
> in your config.h, otherwise there wouldn't be a +HAVE_WC_FUNCS.

No, I don't (unedited variables):

#define HAVE_NCURSES_NCURSES_H 1
#define HAVE_WCHAR_H 1
#define HAVE_WCTYPE_H 1
#define HAVE_WC_FUNCS 1

That's why I tried ncurses.

$ port installed|grep curses
ncurses @5.6_0 (active)
ncursesw @5.6_1 (active)

But this gave me the idea to temporarily uninstall ncurses. Compiled
against just ncursesw,

#define HAVE_NCURSES_H 1

everything is fine, and indeed it stays this
way with ncurses back in place Will report this to the mutt-dev
mailing list.

> > Haven't tried slang yet (partly because I don't like
> > the looks of it), but I will later.
>
> I can't see the slightest difference in mutt between a version with
> slang or a version with ncurses.

I used to see different colors (back on Debian a couple of years ago).

Thank you guys _very_ much for your help!

-André

--
May as well be hung for a sheep as a lamb!
Linkstation/KuroBox/HG/HS/Tera Kernel 2.6/PPC from <http://hvkls.dyndns.org>

On Thu, 29 May 2008, André Berger <andre.berger@web.de> wrote:

> Thomas, I'm very sorry not to have been able to recall your name
> correctly

No worries.

>> No. It should be the other way round. You already have
>> #define HAVE_NCURSESW_NCURSES_H 1
>> in your config.h, otherwise there wouldn't be a +HAVE_WC_FUNCS.
>
> No, I don't (unedited variables):
>
> #define HAVE_NCURSES_NCURSES_H 1
> #define HAVE_WCHAR_H 1
> #define HAVE_WCTYPE_H 1
> #define HAVE_WC_FUNCS 1
>
> That's why I tried ncurses.
>
> $ port installed|grep curses
> ncurses @5.6_0 (active)
> ncursesw @5.6_1 (active)
>
> But this gave me the idea to temporarily uninstall ncurses. Compiled
> against just ncursesw,
>
> #define HAVE_NCURSES_H 1
>
> everything is fine, and indeed it stays this
> way with ncurses back in place Will report this to the mutt-dev
> mailing list.

Fine that everything works now.

JFTR, I did not recall correctly what I said about changing the config
file some posts ago. I found the old source of my mutt 1.5.15 and the
config.h look like this:

#define HAVE_NCURSESW_NCURSES_H 1
/* #undef HAVE_NCURSES_H */
/* #undef HAVE_NCURSES_NCURSES_H */
#define HAVE_WCHAR_H 1
#define HAVE_WCTYPE_H 1
/* #undef HAVE_WC_FUNCS */

The problem was that after the configure run the HAVE_WC_FUNCS was set
and I had to reset this value to undefined to get the Umlauts working
correctly.

Cheers,

Thomas

--
[X] Nail here for new Monitor