[SOLVED] Fixing Icedove character encoding

BLizgreat! · 2016-11-25 05:19:16

Like everyone else is saying looks like an unsupported font/character encoding. Btw: Copy and pasting can cause stuff like this, not all progs support all characters. Had had it happen, create a file in leafpad and then try to open it in another text editor and it'll say nope, don't support blahblahblah or will change the formatting or whatever. Like removing line spacing and crunching all the text together-etc.

Sometimes go to save something I've cut and pasted and whichever program will complain contains unicode characters blahblahblah, save anyway ? Yes/no.

So in this situation, would avoid any copy/paste during testing. I'm stumped on this Sector, never really had a major font/character problem on an OS before. Hmmmmm ... maybe try updating the font cache on the OS ? Terminal "fc-cache -fv" from her user account ... will at least show you all relevant paths on the OS pertaining to fonts, see if the output shows anything unusual. Though guess some apps are going to maintain their own too. Many pull from the systems fonts/characters.

Also a totally out of the air thought. You said she downloads her emails in html mode or something ? Perhaps ... converting them causes issues during the process ? In strict html, there's a buncha characters that someone is supposed to escape to use in the markup, or they can display screwed up ... examples there.

Sorry for whatever reason having a real problem understanding exactly what's going on here.

Last edited by BLizgreat! (2016-11-25 07:33:35)

ohnonot · 2016-11-25 21:02:02

ok, another thought: the language packs for icedove.
i know this was at the beginning of the thread, but i'm too lazy to scroll up...
installing them should add more options to the last 2 dropdowns here, maybe.
maybe you need something else than utf there.

consider setting the fallback encoding for incoming to windows or western.

or try starting icedove with a fresh profile.

consider locale settings for her linux install.

Last edited by ohnonot (2016-11-25 21:03:23)

Sector11 · 2016-11-25 23:02:58

Well, the fact that some emails come in OK, with everything fine and others have problems makes me thing the problem is not here. Why after 12 years would it all of a sudden start? I have a feeling it's at the source of the emails - probably smartphones.

At one time, since starting this thread, I had changed incoming to Western (ISO-8859-1), as it was suggested in one of my searches. Didn't help so they are both back to UTF-8 now

consider locale settings for her linux install.

There is only one install here - we share it. I realize some find that strange and hard to believe but that's the way it is and has been since we got together 15 years ago, starting with Windows, Ubuntu, Xubuntu, #! and now BL.

She uses Icedove and LibreOffice, I use ClawsMail and Medit - and we both use FireFox (I had Palemoon but I lost touch with the repos - must set that up again).

And only one OB setup; her area is a sub-menu off the main branch and most entries are in Spanish for her.

BLizgreat! · 2016-11-26 00:29:36

Alright last stab at this issue. Check this thread out Sector, specifically Toad-Hall posting. Seems to have something in common with the problems you're having. Guy even starts it off with, why after all these years. Might be the fix you're looking for. If not ... I'm at a loss and time to throw in the towel on this. Gotta admit when I can't be of any help eh. Fingers crossed man.

Last edited by BLizgreat! (2016-11-26 00:33:11)

ohnonot · 2016-11-26 08:04:48

^ +1 to that.
i was thinking yesterday about this (i think the config editor solution does the same).

Sector11 · 2016-11-26 14:52:17

@BLwbg and ohnonot

Always amazes me that people talk about [Tools] » [Options] and my Icedove it use [Edit] » [Preferences] - anyway to continue ...

Toad-Hall wrote:

Try this:
1. 'Tools' > 'Options' > 'Display' -> Formatting tab click on 'Advanced' button. UNselect - do not select - "apply the default character encoding to all incoming messages" checkbox. Click on OK
2. 'View' > 'Character encoding' > 'Auto-detect' > 'Universal'
3. Restart Thunderbird.

BTW: Read that entire thread.

#1 As seen and posted here in my post #17 in this thread: [Display] » [Formatting] » [Advanced] = Fonts & Encodings where there is no "apply the default character encoding to all incoming messages" checkbox

#2 [View] » [Character encoding] » [Auto-detect[ » [Universal] also does not exist.

I cannot find: [Character encoding] in either:
[Main Menu] » [View] » or sub-menus
[Edit] » [Preferences] » or any sub-menu
The best I can fined is: [Edit]+[Preferences]+[Display]+[Advanced] as seen in my post #17 above.
Clicking on ALL the options: Universal does not exist.

Now since I'm having problems with "incoming" mail the answer "must be/should be/aught to be" somewhere in:
[Edit]+[Preferences]+[Display]
[Edit]+[Preferences]+[Display]+[Formatting]+[Advanced]
[Edit]+[Preferences]+[Display]+[Tags]
[Edit]+[Preferences]+[Display]+[Advanced]

- not a thing works!

And as yet - not even a nibble on the Mozilla forums, it's like they know there is no fix.

But thanks for trying guys ... Marking: [Unsolvable] as even in the link you gave me that is the consensus.

BLizgreat! · 2016-11-26 15:39:43

Was talking about this post from Toad-Hall.

Could you say what you have hear:
Tools > Options > Advanced > General tab
Click on Config Editor
In top search type:
mailnews.force_charset_override; Value = 'false'
What do you have as the Value..true or false?
If Value = 'True', double click on line to toggle to 'false'
close window - top right X
click on OK to save changes to Options.
Close and reopen Thunderbird.

On this version of thunderbird v 45.4.0 to get to anything even approximating what some people are describing, I have to hit F10 to display a menu, then Edit > Preferences > Advanced tab, then Config Editor is in the bottom right. Under the section titled Advanced Configuration. It's like the about:config setting in Firefox and gives you the this might void your warranty warning, just like in good ole fox. When first entered that search string Toad-Hall gave nothing would come up, so I started hitting the backspace to remove letters from it and the setting he was referring to did pop up. In the thunderbird on this OS, it was set to false default. Though might not be in yours and maybe worth a looksee ?

Fixed the guy in the support forums problems and seems like his were similar to yours ma friend. Anyway, still got fingers crossed.

More babble ... Hey this thing is starting to seem more familiar, now that I know where to find the tweak section. Might start using the sucker myself. Though couple other things if haven't already, I agree if anyone said start in safe mode, there's an option in the main control area of the application to start with plugins disabled and/or trying a new profile on the thing to see if it helps. Would dig around in those about:config settings too and look at whatever isn't set to default value. aka: Things a plugin or whatever has changed from the default setting.

Last edited by BLizgreat! (2016-11-26 15:56:55)

BLizgreat! · 2016-11-26 16:30:37

Oh yeah ... finally dawned on me why peeps keep saying Tools> Options > etc etc. They're using Window$, gnu/Linux is laid out differently.

Sector11 · 2016-11-26 18:43:23

^ AHA! The light finally clicked I'm running Icedove v 1:45.4.0-1 so we are the same in that respect.

Also if you right click to the right of "Inbox" you should be able to turn the Menu Bar on permanently, no more F10:
»

F-10; not to be confused with the A-10 Thunderbolt aka: WartHog aka Hog aka TankBuster

BLizgreat! · 2016-11-26 19:13:40

^Lol ... was throwing me off too Sector. Tools > Options ??? Errrrrrr what ? Shows how often I/we boot into any window$ OS. Yeppers, already set the thing to stay visable, imo not the best default choice for Mozilla to set up the display. Someone new to thunderbird/Icedove, how are they supposed to know to hit F10 ? So gotta know, making any progress sorting out the problem now ?

vll!

Sector11 · 2016-11-26 21:02:27

Progress? - from 'everything' I've read there is no progress. People have had this problem for a long time. reported it and it still exists.

last Windows I opened was in July 2007 when W2K showed a Blue Screen of Death and a message that said "no hard drive found". That has to be one of THE dumbest BSOD messages going!

If there was no HDD, there would be no Windows and therefore no BSOD. DUH!

johnraff · 2016-11-29 07:56:26

Another Icedove user here.
If you'd like to send an email to john [that symbol] bunsenlabs.org that displays wrong in S12's Tbird, I'll check how it looks in mine.

Sector11 · 2016-11-29 19:20:19

^Problem is - icedove sends in plain text. But receives in HTML. How does your Icedove work - text or html?

I will send a couple of emails - one in text and one in html (if I can do that on an individual email. Although the problem is in receiving emails. I've not heard rom anyone saying they cannot read her mails.

Thanks John ... test coming soon.

Here's some results as seen here ... looking at complete hearders:

With these

Content-type: text/html; charset=UTF-8
Content-transfer-encoding: 8bit
--- or ---
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_x5OEVWFM/UeaUI5G4gJWpg)"; charset=UTF-8
Content-transfer-encoding: 8BIT

I see á ñ € ! ¡ ? ¿ just fine!

This in headers creates problems:

<meta http-equiv="Content-Language" content="en-us">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">

This: "se gestionó su" looks like this little diamond with a question mark

johnraff · 2016-11-30 03:02:18

Sector11 wrote:

How does your Icedove work - text or html?

Icedove will send in either plain text or html. You can set individual recipients in the address book to prefer one or the other, along with a default setting.

Incoming mails can be viewed as "plain text", "simple html" or "original html", chosen in View>Message Body As
I usually use "simple html" because it doesn't download any image files, while keeping a more readable layout. Plain text is fine too though.

I will send a couple of emails - one in text and one in html... Although the problem is in receiving emails.

Understood. Since I'd be reading the mail with the same software that your wife uses, it might tell us something if mails that display wrong at your end are OK (or not) here. Awaiting your test mail (not received yet at 11:59 JST).

Sector11 · 2016-11-30 03:43:38

Interesting, they were sent (text):

Date: Tue, 29 Nov 2016 16:30:16 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Icedove/45.4.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi John,

Some test still that has problems comeing from some emails.

se gestionÃ³ su

Ã¡ Ã© Ã Ã³ Ãº Ã½  ... AÃ±o - 75Â°C
Ã  Ã¨ Ã¬ Ã² Ã¹ á»³

Ã¢ Ãª Ã® Ã´ Ã» Å·  This accent is rare:  Ã¶ and Ã¼

Will try and send this same "text" in HTML format.

Encoding for this is UTF-8

and (html)

Date: Tue, 29 Nov 2016 16:46:12 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Icedove/45.4.0
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="------------627A0883CE30CE6B621DD547"

This is a multi-part message in MIME format.
--------------627A0883CE30CE6B621DD547
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi John,

Some test still that has problems coming from some emails.

/se gestionÃ³ su /

Ã¡ Ã© Ã Ã³ Ãº Ã½  ... AÃ±o - 75Â°C
Ã  Ã¨ Ã¬ Ã² Ã¹ á»³
Ã  Ã¨ Ã¬ Ã² Ã¹ á»³
Ã  Ã¨ Ã¬ Ã² Ã¹ á»³
Ã  Ã¨ Ã¬ Ã² Ã¹ á»³

Ã¢ Ãª Ã® Ã´ Ã» Å·  This accent is rare:  Ã¶ and Ã¼ :-)

Will try and sÃ©nd this same "tÃ©xt" Ãn HTML fÃ¶rmÃ¢t.


Goog android doesn't like the accents!
Encoding for this is UTF-8

I notice the source shows the code messed up, but in the emails going out they are fine!

johnraff · 2016-12-01 04:01:40

OK I got six emails, all were marked
Content-Type: text/plain; charset=utf-8; format=flowed
in the source code, and all displayed fine in my Icedove.

Interesting that those characters were not displayed right when viewing the source code of the messages though. The same usually happens with Japanese mails. I assumed that was because they were 'charset="ISO-2022-JP"' not UTF-8, so I'm not sure why the source code of your utf-8 mails doesn't display correctly on my utf-8 system.

Anyway, when displaying the message contents normally, all those characters:

se gestionó su
á é í ó ú ý ... Año - 75°C
à è ì ò ù ỳ
â ê î ô û ŷ This accent is rare: ö and ü

looked fine.

I didn't get any html mails - were they the problem causers? Maybe because you forwarded them the html was stripped out?

Maybe you could send one more, one that definitely fails to display right in your local Icedove? Perhaps send the same mail to your address and mine, so we can be sure that both copies are identical?

Sector11 · 2016-12-01 17:52:31

The problem is 'receiving' some emails in HTML format. Normally all mails from here go out in text formal. I tried re-configuring Icedove to send you some HTML mails but that obviously didn't work.

Now here's the thing. Everything was going along fine until one day - maybe half of the email received were messed up. Since then I have been testing different setting and have it down to the of few are messed up.

28 Nov I received this email and the subject looked like this and in looking at the header I see:

Thread-topic: =?iso-8859-1?Q?Factura_Electr=F3nica_OMINT?=

it 'should' look like this:

Thread-topic: =?iso-8859-1?Q?Factura_Electrónica_OMINT?=

iso-8859-1 encoding stuff - Windows. GRRRRRR!!!

Bigger section:

Date: Mon, 28 Nov 2016 02:39:59 -0300
From: OMINT <xxxxxxxxxxxxxxxxxxx@xxxxxxxxxxxxx>
Subject: =?iso-8859-1?Q?Factura_Electr=F3nica_OMINT?=
To: xxxxxxxxxxxxxx@xxxxxxxxxxxxxxx
Message-id: <FE59E928DF084919B4BA14ACCB22C357@omint.ad>
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.4913
X-Mailer: Microsoft CDO for Windows 2000
Content-type: multipart/mixed;
 boundary="----=_NextPart_000_4CEC6_01D24920.B622C9C0"
Content-class: urn:content-classes:message
Importance: normal
Priority: normal
Thread-topic: =?iso-8859-1?Q?Factura_Electr=F3nica_OMINT?=

Windows 2000 - OH MY!

johnraff · 2016-12-07 08:24:28

(Sector11 sent me some .eml files and a screenshot to look at.)

@Sector11 before we try anything else, I noticed I could trigger the same bad appearance in my Icedove by switching the auto-selected encoding from "Western" to "Unicode", so just check:
View>Text Encoding
and see what it's set at. Those mails you sent me were iso-8859-1 (not a specifically M$ encoding btw) so that above encoding should be set to "Western" not "Unicode". Try changing it. Also check what Auto-Detect points to. I've got it set to "Japanese" but you might be able to choose something more useful.

Sector11 · 2016-12-07 14:13:30

Well, we're getting somewhere. While having an email open that is not showing correctly I see:

View » Text encoding » Unicode

change that to:

View » Text encoding » Western

and the accents instantly pop into place.

So now I pop over to:
Edit » Preferences » Display » Formatting » Advanced and reset Text Encoding defaults:

Outgoing Mail: Western (ISO-8859-1)
Incoming Mail: Western (ISO-8895-1)

Close Icedove and reopen and check the "bad" email.

GRRRR

View » Text encoding » Unicode

What am I doing wrong?

EDIT: Still checking all over the net.

Last edited by Sector11 (2016-12-07 14:15:30)

johnraff · 2016-12-08 04:09:05

And when you open a previously saved .eml file, it's already View » Text encoding » Western?

#21 2016-11-25 05:19:16

Re: [SOLVED] Fixing Icedove character encoding

#22 2016-11-25 21:02:02

Re: [SOLVED] Fixing Icedove character encoding

#23 2016-11-25 23:02:58

Re: [SOLVED] Fixing Icedove character encoding

#24 2016-11-26 00:29:36

Re: [SOLVED] Fixing Icedove character encoding

#25 2016-11-26 08:04:48

Re: [SOLVED] Fixing Icedove character encoding

#26 2016-11-26 14:52:17

Re: [SOLVED] Fixing Icedove character encoding

#27 2016-11-26 15:39:43

Re: [SOLVED] Fixing Icedove character encoding

#28 2016-11-26 16:30:37

Re: [SOLVED] Fixing Icedove character encoding

#29 2016-11-26 18:43:23

Re: [SOLVED] Fixing Icedove character encoding

#30 2016-11-26 19:13:40

Re: [SOLVED] Fixing Icedove character encoding

#31 2016-11-26 21:02:27

Re: [SOLVED] Fixing Icedove character encoding

#32 2016-11-29 07:56:26

Re: [SOLVED] Fixing Icedove character encoding

#33 2016-11-29 19:20:19

Re: [SOLVED] Fixing Icedove character encoding

#34 2016-11-30 03:02:18

Re: [SOLVED] Fixing Icedove character encoding

#35 2016-11-30 03:43:38

Re: [SOLVED] Fixing Icedove character encoding

#36 2016-12-01 04:01:40

Re: [SOLVED] Fixing Icedove character encoding

#37 2016-12-01 17:52:31

Re: [SOLVED] Fixing Icedove character encoding

#38 2016-12-07 08:24:28

Re: [SOLVED] Fixing Icedove character encoding

#39 2016-12-07 14:13:30

Re: [SOLVED] Fixing Icedove character encoding

#40 2016-12-08 04:09:05

Re: [SOLVED] Fixing Icedove character encoding

Board footer