#!/bin/cat
$Id: FAQ.APPEND_errors.txt,v 1.15 2025/09/09 14:55:23 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.APPEND_errors.txt

======================================================================
              Dealing with Imapsync APPEND errors.
======================================================================

Questions answered in this FAQ are:


Q. I have a "could not append" error with 
   "Message contains invalid header" at the end, like:
   Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
   What can I do to transfer these emails?

Q. For some messages, the imapsync log says 
   "could not append", sometimes followed by an explicit message 
   describing what went wrong, or sometimes followed by a not very
   useful message "socket closed while reading data from server"
   What can I do?

Q. The append error message is "NO Message contains NUL characters"
   What can I do?

Q. I have this APPEND error:
   Err 98/100: - msg INBOX/277 {29390} could not append ... Flags:[\Seen blob ] ) ...: 280 BAD Error in IMAP command APPEND: 8bit data in atom
   What can I do?

Now the questions again with their answers.


======================================================================
Q. I have a "could not append" error with 
   "Message contains invalid header" at the end, like:
   Err 1/20: - msg INBOX/6 {40666} could not append ... NO Message contains invalid header
   What can I do to transfer these emails?


R0. Append in the IMAP protocol is the command to add a message to the folder.
    The error string "NO. Message contains invalid header" comes from the 
    destination IMAP server at host2, it doesn't like the message and rejects it.
   
R1. Header lines must be in 7-bit encoding.
    If they aren't in 7-bit, the IMAP server can refuse them as they break 
    the IMAP RFC specification.

    A solution is to transform all 8-bit characters to the Z character with:
   
      imapsync ... --regexmess "tr [\x80-\xff] [Z]"

    Caveat: this transformation transforms both the email header and its body.
    It's not perfect, but it is better than no message transferred.
    So consider using it in a second pass, not the first, as 8-bit 
    characters in the body message are allowed, and changing them when
    not needed is not a good thing.


R2. Some crappy email systems, like virus checkers or any other crappy 
    software tool dealing with your messages, can add headers to tell 
    the world that they've done a great job. But sometimes they didn't,
    and they fucked up your messages by not respecting the standard
    and adding ugly non-RFC-compliant headers.
    
    To remove an ugly header, let's call it "X-Spam-Report", that spreads over 
    several lines beginning or, that's the ugly part, not beginning with a space:

    imapsync ... --regexmess 's{X-Spam-Report:.*?\n(^[a-zA-Z0-9\-]+:|^\r?\n)}{$1}xms'

    Thanks to Damien SAUTEREAU for reporting and solving this issue.
    

======================================================================
Q. For some messages, the imapsync log says 
   "could not append", sometimes followed by an explicit message 
   describing what went wrong, or sometimes followed by a not very
   useful message "socket closed while reading data from server"
   What can I do?

R0. Append errors are host2 problems, append is the imap term to copy 
    a message to the destination account. 
    A problem is that the associated "socket closed ..." error message happens
    for several different issues. So I list here several potential issues
    and their solutions, if they exist.
    R1 deals with too-long lines in messages on Windows.
    R2 deals with too-long lines in messages on Unix.
    R3 deals with the quota reached.

R1. On Windows, add --regexmess "s,(.{9900}),$1\r\n,g"

Some messages have too long lines; for example, 
Exchange supports only 9900 characters per line.
Use this option to add "new line" characters (also called CRLF)
to wrap lines longer than 9900 characters. 
The regex means "add one CRLF every 9900".

  imapsync.exe ... --regexmess "s,(.{9900}),$1\r\n,g"

R2. On Unix, add --pipemess "reformime -r7". The command reformime
    usually belongs to the package called "maildrop".

  imapsync ... --pipemess "reformime -r7"

I reproduce here the "reformime" manual part explaining what  
the option "-r7" does:

$ man reformime |more
REFORMIME(1)                Double Precision, Inc.                REFORMIME(1)

NAME
       reformime - MIME E-mail reformatting tool

SYNOPSIS
       reformime [options...]

DESCRIPTION
       reformime is a utility for reformatting MIME messages.

       Generally, reformime expects to see an RFC 2045[1] compliant message on
       standard input
...
OPTIONS
...
       -r
           Rewrite message, adding or standardizing RFC 2045[1] MIME headers.

       -r7
           Like -r but also convert 8-bit-encoded MIME sections to
           quoted-printable.
...
   Adding RFC 2045 MIME headers
       The -r option performs the following actions:

       If there is no Mime-Version:, Content-Type:, or
       Content-Transfer-Encoding: header, reformime adds one.

       If the Content-Transfer-Encoding: header contains 8bit or raw, but only
       seven-bit data is found, reformime changes the
       Content-Transfer-Encoding header to 7-bit.

       -r7 does the same thing, but also converts 8-bit-encoded content that
       contains eight-bit characters to quoted-printable encoding.

       
R3. Look for the quota limit on host2. If it is the current mailbox size, don't
    search elsewhere. Increase the quota on the destination account.

======================================================================
Q. The append error message is "NO Message contains NUL characters"
   What can I do?

R. Use:

  imapsync ...  --skipmess "m/(\x00)+\Z/"

The option --skipmess skips messages matching a pattern; the 
messages are not synced, you just avoid the error messages 
concerning those messages.

If you want to sync them, use instead:

  imapsync ...  --regexmess "s/(\x00)+\Z//g"


=======================================================================
Q. I have this APPEND error:
   Err 98/100: - msg INBOX/277 {29390} could not append ... Flags:[\Seen blob ] ) ...: 280 BAD Error in IMAP command APPEND: 8bit data in atom
   What can I do?

A. Use this to remove all 8-bit characters from the flags:

  imapsync ... --regexflag "tr [\x80-\xff] []d"

======================================================================
======================================================================
