Exhange 2007 to Cyrus 2.3.11

Hello,

I am trying to migrate a few mailboxes from Exchange 2007 to Cyrus
2.3.11 using imapsync and I am getting the following error after a few
messages are copied to the mailbox on the Cyrus server.

Here is the command I ran:
imapsync --maxage 1 --noauthmd5 --host1 t5mapexch01 --user1 "user1" --
password1 "xxx" --host2 mailhost --user2 "user2" --
password2 "xxx" --regextrans2 's/INBOX.INBOX/INBOX/'

Here is the error I am receiving.

+ NO msg #128071 [+MkWJksVVyOj6fGpAEeh3A:1497] in INBOX
+ Copying msg #128071:1497 to folder INBOX
flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
parse_headers want an ARRAY ref

Mike Little writes:

> Hello,
>
> I am trying to migrate a few mailboxes from Exchange 2007 to Cyrus
> 2.3.11 using imapsync and I am getting the following error after a few
> messages are copied to the mailbox on the Cyrus server.
>
> Here is the command I ran:
> imapsync --maxage 1 --noauthmd5 --host1 t5mapexch01 --user1 "user1" --
> password1 "xxx" --host2 mailhost --user2 "user2" --
> password2 "xxx" --regextrans2 's/INBOX.INBOX/INBOX/'
>
> Here is the error I am receiving.
>
> + NO msg #128071 [+MkWJksVVyOj6fGpAEeh3A:1497] in INBOX
> + Copying msg #128071:1497 to folder INBOX
> flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> parse_headers want an ARRAY ref

Bug in imapsync, which I presume is a Perl script. If this is the current
version of imapsync, report this to imapsync's author.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQBH/qEhx9p3GYHlUOIRAv2LAJ9BJhsOkpvv6k+4Nk5TQFUuMGW5YAC fYGgs
FYQOKlndivaLvf960ga1LE0=
=35wx
-----END PGP SIGNATURE-----

Hello,

> + NO msg #128071 [+MkWJksVVyOj6fGpAEeh3A:1497] in INBOX
> + Copying msg #128071:1497 to folder INBOX
> flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> parse_headers want an ARRAY ref

This error message appears only with imapsync 1.249
For now, the error message is false but it means there was
a problem during the transfer of a message. I'll try to fix that.

The strange thing is that we should see some output like this:
From Folder [...
To Folder [...
++++ From [...] Parse 1 ++++

between
flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
and
parse_headers want an ARRAY ref

Since we didn't see it, it's pure mystery for me.

On Apr 11, 4:24 am, Gilles Lamiral <gilles.lami...@gmail.com> wrote:
> Hello,
>
> > + NO msg #128071 [+MkWJksVVyOj6fGpAEeh3A:1497] in INBOX
> > + Copying msg #128071:1497 to folder INBOX
> > flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> > parse_headers want an ARRAY ref
>
> This error message appears only with imapsync 1.249
> For now, the error message is false but it means there was
> a problem during the transfer of a message. I'll try to fix that.
>
> The strange thing is that we should see some output like this:
> From Folder [...
> To Folder [...
> ++++ From [...] Parse 1 ++++
>
> between
> flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> and
> parse_headers want an ARRAY ref
>
> Since we didn't see it, it's pure mystery for me.

I'm also getting the issue every once in awhile. and my output looks
pretty much the same

From state Authenticated
To state Authenticated
Using [Public Folders/] given by --prefix1
From separator and prefix : [/][Public Folders/]
To separator and prefix : [/][]
++++ Calculating sizes ++++
From Folder [Public Folders/zzNonSpam] Size: 163574
Messages: 59
Total size: 163574
Total messages: 59
Time : 1 s
++++ Calculating sizes ++++
To Folder [zzNonSpam] Size: 517214
Messages: 162
Total size: 517214
Total messages: 162
Time : 1 s
++++ Listing folders ++++
From folders list : [Public Folders/zzNonSpam]
To folders list : [zzNonSpam]
++++ Looping on each folder ++++
From Folder [Public Folders/zzNonSpam]
To Folder [zzNonSpam]
++++ From [Public Folders/zzNonSpam] Parse 1 ++++
++++ To [zzNonSpam] Parse 1 ++++
++++ Verifying [Public Folders/zzNonSpam] -> [zzNonSpam] ++++
+ NO msg #89653 [TlmtosMhcnozZ3n593b1cA:1070] in zzNonSpam
+ Copying msg #89653:1070 to folder zzNonSpam
flags from : [\Deleted]["14-Apr-2008 12:17:43 -0500"]
Copied msg id [89653] to folder zzNonSpam msg id
[Mail::IMAPClient=HASH(0x91b3a2c)]
Deleting msg #89653 in folder Public Folders/zzNonSpam
... (a few more messages like above)
+ NO msg #89659 [XZ3z7aai3zeCd05Ffm8gjQ:1746] in zzNonSpam
+ Copying msg #89659:1746 to folder zzNonSpam
flags from : [\Deleted]["14-Apr-2008 13:03:41 -0500"]
Copied msg id [89659] to folder zzNonSpam msg id
[Mail::IMAPClient=HASH(0x91b3a2c)]
Deleting msg #89659 in folder Public Folders/zzNonSpam
+ NO msg #89660 [W9lcx4C1pBX6kZK87A1jMw:5187] in zzNonSpam
+ Copying msg #89660:5187 to folder zzNonSpam
flags from : [()]["14-Apr-2008 13:52:25 -0500"]
parse_headers want an ARRAY ref

I get this periodically with certain emails I'm moving. What if
anything can I provide / do troubleshoot this myself ?

Hi,

please have a look at KB 937359 regarding Exchange and capability for
domain/admin/user. If you want to use imapsync with an admin account
this could be a problem!? --authuser doesn't work with exchange :-(
Also Exchange has a bug since SP1 which adds a Received entry to the
header with the date of the migration. This makes all mails appear as
recently received in Outlook etc. I don't know if that affects you but
just keep in mind to have a look at the headers (before and after).
Maybe they are modified as well (outgoing).

Fabian

On Apr 11, 10:24 am, Gilles Lamiral <gilles.lami...@gmail.com> wrote:
> Hello,
>
> > + NO msg #128071 [+MkWJksVVyOj6fGpAEeh3A:1497] in INBOX
> > + Copying msg #128071:1497 to folder INBOX
> > flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> > parse_headers want an ARRAY ref
>
> This error message appears only with imapsync 1.249
> For now, the error message is false but it means there was
> a problem during the transfer of a message. I'll try to fix that.
>
> The strange thing is that we should see some output like this:
> From Folder [...
> To Folder [...
> ++++ From [...] Parse 1 ++++
>
> between
> flags from : [\Seen]["09-Apr-2008 10:27:35 -0400"]
> and
> parse_headers want an ARRAY ref
>
> Since we didn't see it, it's pure mystery for me.

I am having the same issue. In this case it is preventing a large
migration, since when the error occurs (always with the same specific
message) imapsync exits and no more messages are copied. I tried to
copy the troublesome ones by hand the first few times it happened but
there are just too many for this to be practicle. If there are any
tips on how to proceed, or I can provide any debugging info please let
me know.

Here is the command I am using (imapsync 1.250)
imapsync --host1 host --port1 143 --user1 user --passfile1 pass1 --
host2 host2 --port2 143 --user2 user1 --passfile2 file2 --noauthmd5 --
include '^BillingResponse' --regextrans2 's#(.*)#INBOX/WINNIPEG-COPY/
$1#' --useheader 'Message-ID' --skipsize --debug

Here is the output:

Turned ON syncinternaldates, will set the internal dates on host2 same
as host1.
TimeZone :[BST]
Will try to use LOGIN authentication on host1
Will try to use LOGIN authentication on host2
From imap server [] port [143] user [t]
To imap server [] port [143] user [t]
Banner : * OK winnipeg Cyrus IMAP4 v2.1.18-IPv6-Debian-2.1.18-1 server
ready
Host 192.168.7.3 says it has NO CAPABILITY for AUTHENTICATE LOGIN
Success login on [] with user [t] auth [LOGIN]
Banner : * OK Microsoft Exchange Server 2007 IMAP4 service ready
Host 192.168.6.24 says it has NO CAPABILITY for AUTHENTICATE LOGIN
Success login on [] with user [t] auth [LOGIN]
From capability : ID QUOTA ANNOTATEMORE NAMESPACE MULTIAPPEND IDLE
UNSELECT SORT NO_ATOMIC_RENAME UIDPLUS THREAD=REFERENCES IMAP4
IMAP4REV1 AUTH=GSSAPI MAILBOX-REFERRALS THREAD=ORDEREDSUBJECT ACL
LITERAL+ CHILDREN
To capability : LITERAL+ NAMESPACE STARTTLS AUTH=GSSAPI AUTH=NTLM
IDLE IMAP4REV1 IMAP4
From state Authenticated
To state Authenticated
Including folders matching pattern '^BillingResponse': BillingResponse
From separator and prefix : [/][]
To separator and prefix : [/][]
++++ Calculating sizes ++++
From Folder [BillingResponse] Size: 113630
Messages: 21
Total size: 113630
Total messages: 21
Time : 1 s
++++ Calculating sizes ++++
To Folder [INBOX/WINNIPEG-COPY/BillingResponse] Size: 47306
Messages: 8
Total size: 47306
Total messages: 8
Time : 0 s
++++ Listing folders ++++
From folders list : [BillingResponse]
To folders list : [INBOX/WINNIPEG-COPY/BillingResponse]
++++ Looping on each folder ++++
From Folder [BillingResponse]
To Folder [INBOX/WINNIPEG-COPY/BillingResponse]
++++ From [BillingResponse] Parse 1 ++++
++++ To [INBOX/WINNIPEG-COPY/BillingResponse] Parse 1 ++++
++++ Verifying [BillingResponse] -> [INBOX/WINNIPEG-COPY/
BillingResponse] ++++
+ NO msg #5 [TDKAAoS3tODbCCBAfgk6sw] in INBOX/WINNIPEG-COPY/
BillingResponse
+ Copying msg #5:4353 to folder INBOX/WINNIPEG-COPY/BillingResponse
flags from : [\Seen NonJunk]["11-Jan-2008 10:47:02 +0100"]
parse_headers want an ARRAY ref

The relevant debug output is:
F message content ended on previous line
================================================== ==============================
internal date from 1: [11-Jan-2008 09:47:02 +0000]
internal date from 1: ["11-Jan-2008 10:47:02 +0100"] (fixed)
flags from : [\Seen NonJunk]["11-Jan-2008 10:47:02 +0100"]
parse_headers want an ARRAY ref


Cheers,

Robin

jweinshe@gmail.com wrote:
> I'm also getting the issue every once in awhile. and my output looks
> pretty much the same

Upgrade to 1.250 which will survive this problem.

The problem is (I guess) that you run with --useheader Message-ID
and the message in question does not have a Message-ID and therefore
there are no headers for imapsync to check. The net result will be that
these messages are not (ever) synchronized over. You have to take more
headers into account.

I have the same problem trying to synchronize TO Exchange, and that
is somewhat more problematic, as Exchange does some nasty "normalisation"
of headers, reorders them, changes case, timezone normalizes dates,
etc ad absurdum. All in all it is difficult to come up with a useful
header set to replace Message-ID.

Does anybody have good results with any tool for conversion to
Exchange?

/Niels

--
Niels Baggesen -- @home -- Århus -- Denmark -- niels@baggesen.net
The purpose of computing is insight, not numbers -- R W Hamming