Re: Questions on Keywords - IMAP

On Tue, 14 Mar 2006, NM Public wrote:
>> Keywords that do not begin with $ are all for the human user's exclusive
>> use.
> I didn't know that that was what the $ means -- is this documented somewhere?

It's a convention, and not enforced in the protocol. The idea was that
all keywords which would have special meaning to some software would be
start with $, and be registered someplace. Similarly, all keywords which
do not start with $ would be solely for humans.

Alexey's document was supposed to detail this. I'm distressed to see that
it's gone. I'll ask Alexey about it when I see him in a couple of weeks.

> Note that some mailbox formats, for example mbox when used with a UW IMAP
> server, store the metadata in the headers.

That doesn't matter. As with Status: and other metadata headers, it is
removed from the header that is presented to the user. It may appear in
the header in the file, but the user's question was what he sees when
viewing full headers. As far as the user is concerned, it doesn't exist
in the header.

> Note that keywords can be "transmitted" with a message if the message is
> transported via the IMAP protocol, e.g., from one IMAP mailbox to another.

That wasn't the user's question.

> Also note that delivery agents, such as Procmail and Sieve, can be used to
> set keywords. I do this extensively with Sieve's 'addflag' command.

That's only if the delivery agent knows how to set metadata, and/or
communicates with a delivery agent that knows. That is not always the
case, and in many cases is not the case.

I would definitely NOT depend upon a procmail rule that manually adds the
header line to a traditional UNIX mailbox file, because that can get
filtered out as part of a system manager reconfiguration of delivery.

Going through Sieve is (much) better, but that assumes that Sieve is
installed. That's a big assumption.

I'm glad that it works for you (it was certainly intended to!); but you
really need to emphasize the "power user" nature of this in most
installations. Only on a few installations, which have done the necessary
technical groundwork, is the capability available in a reliable
user-accessible way. On most installations, it is either unavailable or
available only via kludge.

> Whenever I need to investigate something like this, which I recently did with
> the Opera 9 Preview, I use the client (e.g., Opera or Outlook) to access an
> mbox-formatted mailbox on a UW IMAP server and set each tag/category/flag on
> a separate message. Then I quit the client and ssh to the UW IMAP server and
> look at the raw mbox (e.g., using 'less mboxname'. If IMAP keywords are being
> used, they should be in X-Keywords headers.

This is one of those kludges. It assumes that the site offers shell
access to the IMAP server (a big assumption) and that the site won't
change mailbox formats.

Don't get me wrong. I urge you to continue passing on this information;
it's good and valuable information for the community. You're doing a
fantastic service too! [And in case you don't hear it enough, "Thank
you!!!!"]

But please remember to have suitable cautionary notes. What works for
you, a power user with the technical skills to handle a curve ball tossed
your way, may not work as easily for a novice.

I don't want people to get the idea that they can copy a procmail script
that insert X-Keywords into a header and it will just work as
advertised...

The good news is that with greater use for keywords, there's greater
demand. Thus, we can expect that IMAP server implementors and site
managers will be more likely to put Sieve and other official mechanisms in
place for keywords to be set during delivery.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin wrote:
> On Tue, 14 Mar 2006, NM Public wrote:
>
>>Note that keywords can be "transmitted" with a message if the message is
>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>
> That wasn't the user's question.

Actually, I'm not sure.

Does "transmitting" mean "sending via SMTP" ?

Couldn't keywords be sent with a X-Keyword header and eventually a
content-disposition header ?

Or perhaps thanks to http://www.ietf.org/rfc/rfc3458.txt ?

Alexey once mentioned : "There is currently no standard for forwarding
keywords over SMTP, however some servers like Cyrus LMTP support an
extension for doing that" -
http://joseph.randomnetworks.com/arc.../#comment-2220

There's some demand to send mails with keywords : "But when am I going
to be able to do this with email, in a way that is recognized by the
recipient and the sender?" -
http://www.choiresicha.com/archives/000578.html and that kind of tool
might be very useful in a corporate/biz environment.

--
kael

kael writes:

> Mark Crispin wrote:
>> On Tue, 14 Mar 2006, NM Public wrote:
>>
>>>Note that keywords can be "transmitted" with a message if the message is
>>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>>
>> That wasn't the user's question.
>
> Actually, I'm not sure.
>
> Does "transmitting" mean "sending via SMTP" ?

No, it means "copied or uploaded via IMAP, to some mail folder".

> Couldn't keywords be sent with a X-Keyword header and eventually a
> content-disposition header ?

You may include whatever header pleases you, as part of a message that you
wish to send via SMTP. When the message gets delivered to the recipient,
if this header remains intact, and the message is opened by the UW-IMAP
server, it's likely that this header will be processed accordingly, and the
message will be marked with the given keywords.

However, if the message is opened by some other mail handling software, the
X-Keyword: header will likely to be ignored completely.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQBEF0x1x9p3GYHlUOIRAuZtAJ0SmTnIFgWrlVUj+iQgc4tUmXsLagCfXIJo
vLDdc71Qo/ee3V+wt1V9Oc0=
=hop3
-----END PGP SIGNATURE-----

Sam wrote:
> kael writes:
>>Mark Crispin wrote:
>>>On Tue, 14 Mar 2006, NM Public wrote:
>>>
>>>>Note that keywords can be "transmitted" with a message if the message is
>>>>transported via the IMAP protocol, e.g., from one IMAP mailbox to another.
>>>
>>>That wasn't the user's question.
>>
>>Actually, I'm not sure.
>>
>>Does "transmitting" mean "sending via SMTP" ?
>
> No, it means "copied or uploaded via IMAP, to some mail folder".

The original question is slightly ambiguous.

>>Couldn't keywords be sent with a X-Keyword header and eventually a
>>content-disposition header ?
>
> You may include whatever header pleases you, as part of a message that you
> wish to send via SMTP. When the message gets delivered to the recipient,
> if this header remains intact, and the message is opened by the UW-IMAP
> server, it's likely that this header will be processed accordingly, and the
> message will be marked with the given keywords.
>
> However, if the message is opened by some other mail handling software, the
> X-Keyword: header will likely to be ignored completely.

Couldn't headers for keywords be comparable to those from mailing-lists
as defined by http://www.ietf.org/rfc/rfc2369.txt ?

Don't IMAP client deal with those headers in a specific manner thanks to
the ENVELOPE structure ?

--
kael

On Tue, 14 Mar 2006, Sam wrote:
> > Couldn't keywords be sent with a X-Keyword header and eventually a
> > content-disposition header ?
> You may include whatever header pleases you, as part of a message that you
> wish to send via SMTP. When the message gets delivered to the recipient, if
> this header remains intact, and the message is opened by the UW-IMAP server,
> it's likely that this header will be processed accordingly, and the message
> will be marked with the given keywords.

UW imapd's affiliated mail delivery tools, and several unrelated mail
delivery tools, go to some length (and in the future will do more) to
discard metadata headers in SMTP-incoming messages that are destined for
traditional UNIX format messages.

Although it is still possible to sneak a metadata header today on some
sites, it likely will not still work tommorrow. There are too many
attacks caused by allowing sender-specified metadata. Sites are
increasingly deploying mail delivery tools that block such headers.

Nor do you want to allow metadata headers to be transmitted via SMTP. If
that happened, then anybody could do all sorts of things to your mail
viewing environment just by sending you a message.

Note that this problem is specific to traditional UNIX format files, which
by their nature store metadata in the headers. Most IMAP servers,
including UW imapd, support other formats which do not store metadata in
the headers. Some IMAP servers do not support traditional UNIX format at
all.

> However, if the message is opened by some other mail handling software,
> the X-Keyword: header will likely to be ignored completely.

It's not a software issue; it's a mailbox format issue.

Thus, a more accurate statement is:

However, unless the message is delivered to a traditional UNIX format
mailbox file, the X-Keyword header will likely be ignored complete.
Other formats, which do not include metadata in headers, are completely
immune from this attack.

The bottom line is that playing with metadata headers such as X-Keyword is
like playing with black magic. It won't always work, it's likely to
backfire on you, and it may incur the wrath of the sorcerer... ;-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

kael writes:

> Sam wrote:
>
>>>Couldn't keywords be sent with a X-Keyword header and eventually a
>>>content-disposition header ?
>>
>> You may include whatever header pleases you, as part of a message that you
>> wish to send via SMTP. When the message gets delivered to the recipient,
>> if this header remains intact, and the message is opened by the UW-IMAP
>> server, it's likely that this header will be processed accordingly, and the
>> message will be marked with the given keywords.
>>
>> However, if the message is opened by some other mail handling software, the
>> X-Keyword: header will likely to be ignored completely.
>
> Couldn't headers for keywords be comparable to those from mailing-lists
> as defined by http://www.ietf.org/rfc/rfc2369.txt ?

Show me a meaningful RFC that defines a header for holding keywords.

> Don't IMAP client deal with those headers in a specific manner thanks to
> the ENVELOPE structure ?

The RFC2369 headers are not returned by ENVELOPE. Nor BODYSTRUCTURE.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQBEF2MKx9p3GYHlUOIRAh76AJ9Nmcr5+vHEpOeSXa/w5hEzET8cagCeJeFi
06jORUTyanlolO2tlhrvNrA=
=RCDi
-----END PGP SIGNATURE-----

On Wed, 15 Mar 2006, kael wrote:
> Couldn't headers for keywords be comparable to those from mailing-lists as
> defined by http://www.ietf.org/rfc/rfc2369.txt ?

IMAP keywords have nothing in common with message headers. It is simply
an artifact of traditional UNIX mailbox format that they are stored in the
header at all.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Sur 2006-03-14, Mark Crispin skribis:
> On Tue, 14 Mar 2006, NM Public wrote:
>>
>>> Keywords that do not begin with $ are all for the human
>>> user's exclusive use.
>>>
>> I didn't know that that was what the $ means -- is this
>> documented somewhere?
>
> It's a convention, and not enforced in the protocol. The idea
> was that all keywords which would have special meaning to some
> software would be start with $, and be registered someplace.
> Similarly, all keywords which do not start with $ would be
> solely for humans.
>
> Alexey's document was supposed to detail this. I'm distressed
> to see that it's gone. I'll ask Alexey about it when I see him
> in a couple of weeks.

OK, thanks for the details about the leading '$' -- this explains
why I felt like Alexey and I were not communicating well when I
had email with him about the "Common IMAP Keywords" draft. The
reason I felt that way was because we in fact WERE
miscommunicating. I was making suggestions for keywords that
users (i.e. humans) would use, but that's not what those
leading-$ keywords are for!

Maybe we need two types of common IMAP keywords:

Common User IMAP Keywords
Common Software IMAP Keywords


>> Note that keywords can be "transmitted" with a message if the
>> message is transported via the IMAP protocol, e.g., from one
>> IMAP mailbox to another.
>
> That wasn't the user's question.

The user's question was this: "How are keywords transmitted along
with a message?"

To me, it's still not clear what he was asking and I was simply
trying to make it clear that IMAP keywords should be preserved
when a message is "transmitted" via the IMAP protocol. As long as
the target IMAP system supports IMAP keywords, of course.


>> Also note that delivery agents, such as Procmail and Sieve,
>> can be used to set keywords. I do this extensively with
>> Sieve's 'addflag' command.
>
> That's only if the delivery agent knows how to set metadata,
> and/or communicates with a delivery agent that knows. That is
> not always the case, and in many cases is not the case.
>
> I would definitely NOT depend upon a procmail rule that
> manually adds the header line to a traditional UNIX mailbox
> file, because that can get filtered out as part of a system
> manager reconfiguration of delivery.
>
> Going through Sieve is (much) better, but that assumes that
> Sieve is installed. That's a big assumption.
>
> I'm glad that it works for you (it was certainly intended to!);
> but you really need to emphasize the "power user" nature of
> this in most installations. Only on a few installations, which
> have done the necessary technical groundwork, is the capability
> available in a reliable user-accessible way. On most
> installations, it is either unavailable or available only via
> kludge.

OK, I'll update my Procmail page to make it clear that this is a
"power user" feature and won't necessarily work in future
versions of UW IMAP.

One of the reasons that I created my IMAP Service Providers page
was to get users to demand tools like Sieve and its 'addflag'
extension. If Sieve is available everywhere, more IMAP clients
will start supporting Sieve and the ManageSieve protocol, and
user's will be able to easily control their own server-side
filtering, especially greenlisting (aka whitelisting).

[deleted]

> Don't get me wrong. I urge you to continue passing on this
> information; it's good and valuable information for the
> community. You're doing a fantastic service too! [And in case
> you don't hear it enough, "Thank you!!!!"]

Thank YOU Mark for all the work you've done on IMAP. And thanks
for the thanks too!


> But please remember to have suitable cautionary notes. What
> works for you, a power user with the technical skills to handle
> a curve ball tossed your way, may not work as easily for a
> novice.
>
> I don't want people to get the idea that they can copy a
> procmail script that insert X-Keywords into a header and it
> will just work as advertised...
>
> The good news is that with greater use for keywords, there's
> greater demand. Thus, we can expect that IMAP server
> implementors and site managers will be more likely to put Sieve
> and other official mechanisms in place for keywords to be set
> during delivery.

Yes, yes, this is good news. To push it all forward, please
everyone demand support for IMAP keywords, Sieve, and ManageSieve
from your IMAP service providers, your IMAP desktop clients, and
your IMAP web-based clients!

Nancy
posted to c.m.p and c.m.i, and Bcc'd to Alexey Melnikov

--
Nancy McGough ~ <http://www.ii.com> ~ <http://deflexion.com>

On Wed, 15 Mar 2006, NM Public wrote:
> Maybe we need two types of common IMAP keywords:
> Common User IMAP Keywords
> Common Software IMAP Keywords

I fail to understand your point. If something is a "Common User IMAP
Keyword", what does that mean as opposed to, say, user keywords of
PurplePeopleEater or Blurdybloop?

By design, user keywords convey no meaning other than to the human user
who created them. If the keyword is listed in a document, it presumably
does so for the benefit of software which interprets it. An entire space
of keywords, starting with $, was grafted from the user keyword space for
that purpose (in addition to the IMAP protocol reserved space of keywords
that start with \).

I personally would be very unhappy if some MUA were to interpret my user
keywords and to take its own unilateral action based upon what it thought
should be done with them.

Since user keyword names are arbitrary and have no meaning to software,
the only thing that software can do with such names is present them to the
user as being associated with the message, and to allow users to set
(which may imply creating a new user keyword) and unset user keywords in a
message.

> OK, I'll update my Procmail page to make it clear that this is a "power user"
> feature and won't necessarily work in future versions of UW IMAP.

Just to be sure that you understand; the "power user" feature that is
likely to vanish is that of inserting an X-Keywords header into the
message. It assumes that the target mailbox is in traditional UNIX
mailbox format, and that the means of insertion successfully evaded the
code that defends against SMTP senders including metadata headers in the
message.

The ability to set keywords in a message during delivery via some
mechanism such as Sieve's addflag is also a "power user" feature, but is
NOT likely to go away. In fact, this is the mechanism that users SHOULD
use instead of hacking internal message metadata.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.