Package lamson :: Module encoding
[hide private]
[frames] | no frames]

Module encoding

source code


Lamson takes the policy that email it receives is most likely complete garbage 
using bizarre pre-Unicode formats that are irrelevant and unnecessary in today's
modern world.  These emails must be cleansed of their unholy stench of
randomness and turned into something nice and clean that a regular Python
programmer can work with:  unicode.

That's the receiving end, but on the sending end Lamson wants to make the world
better by not increasing the suffering.  To that end, Lamson will canonicalize
all email it sends to be ascii or utf-8 (whichever is simpler and works to
encode the data).  When you get an email from Lamson, it is a pristine easily
parseable clean unit of goodness you can count on.

To accomplish these tasks, Lamson goes back to basics and assert a few simple
rules on each email it receives:

1) NO ENCODING IS TRUSTED, NO LANGUAGE IS SACRED, ALL ARE SUSPECT.
2) Python wants Unicode, it will get Unicode.
3) Any email that CANNOT become Unicode, CANNOT be processed by Lamson or
Python.
4) Email addresses are ESSENTIAL to Lamson's routing and security, and therefore
will be canonicalized and properly encoded.
5) Lamson will therefore try to "upgrade" all email it receives to Unicode
internally, and cleaning all email addresses.
6) It does this by decoding all codecs, and if the codec LIES, then it will
attempt to statistically detect the codec using chardet.
7) If it can't detect the codec, and the codec lies, then the email is bad.
8) All text bodies and attachments are then converted to Python unicode in the
same way as the headers.
9) All other attachments are converted to raw strings as-is.

Once Lamson has done this, your Python handler can now assume that all
MailRequest objects are happily unicode enabled and ready to go.  The rule is:

    IF IT CANNOT BE UNICODE, THEN PYTHON CANNOT WORK WITH IT.

On the outgoing end (when you send a MailResponse), Lamson tries to create the
email it wants to receive by canonicalizing it:

1) All email will be encoded in the simplest cleanest way possible without
losing information.
2) All headers are converted to 'ascii', and if that doesn't work, then 'utf-8'.
3) All text/* attachments and bodies are converted to ascii, and if that doesn't
work, 'utf-8'.
4) All other attachments are left alone.
5) All email addresses are normalized and encoded if they have not been already.

The end result is an email that has the highest probability of not containing
any obfuscation techniques, hidden characters, bad characters, improper
formatting, invalid non-characterset headers, or any of the other billions of
things email clients do to the world.  The output rule of Lamson is:

    ALL EMAIL IS ASCII FIRST, THEN UTF-8, AND IF CANNOT BE EITHER THOSE IT WILL
    NOT BE SENT.

Following these simple rules, this module does the work of converting email
to the canonical format and sending the canonical format.  The code is 
probably the most complex part of Lamson since the job it does is difficult.

Test results show that Lamson can safely canonicalize most email from any
culture (not just English) to the canonical form, and that if it can't then the
email is not formatted right and/or spam.

If you find an instance where this is not the case, then submit it to the
project as a test case.

Classes [hide private]
  EncodingError
Thrown when there is an encoding error.
  MailBase
MailBase is used as the basis of lamson.mail and contains the basics of encoding an email.
  MIMEPart
A reimplementation of nearly everything in email.mime to be more useful for actually attaching things.
Functions [hide private]
 
VALUE_IS_EMAIL_ADDRESS(v)
@
source code
 
from_message(message)
Given a MIMEBase or similar Python email API message object, this will canonicalize it and give you back a pristine MailBase.
source code
 
to_message(mail)
Given a MailBase message, this will construct a MIMEPart that is canonicalized for use with the Python email API.
source code
 
to_string(mail, envelope_header=False)
Returns a canonicalized email string you can use to send or store somewhere.
source code
 
from_string(data)
Takes a string, and tries to clean it up into a clean MailBase.
source code
 
to_file(mail, fileobj)
Writes a canonicalized message to the given file.
source code
 
from_file(fileobj)
Reads an email and cleans it up to make a MailBase.
source code
 
normalize_header(header) source code
 
parse_parameter_header(message, header) source code
 
decode_message_body(mail, message) source code
 
properly_encode_header(value, encoder, not_email)
The only thing special (weird) about this function is that it tries to do a fast check to see if the header value has an email address in it.
source code
 
header_to_mime_encoding(value, not_email=False) source code
 
header_from_mime_encoding(header) source code
 
guess_encoding_and_decode(original, data, errors='strict') source code
 
attempt_decoding(charset, dec) source code
 
apply_charset_to_header(charset, encoding, data) source code
 
_match(data, pattern, pos) source code
 
_tokenize(data, next) source code
 
_scan(data) source code
 
_parse_charset_header(data) source code
 
properly_decode_header(header) source code
Variables [hide private]
  DEFAULT_ENCODING = 'utf-8'
  DEFAULT_ERROR_HANDLING = 'strict'
  CONTENT_ENCODING_KEYS = set(['Content-Disposition', 'Content-T...
  CONTENT_ENCODING_REMOVED_PARAMS = ['boundary']
  REGEX_OPTS = 10
  ENCODING_REGEX = re.compile(r'(?im)=\?([a-z0-9-]+?)\?([bq])\?')
  ENCODING_END_REGEX = re.compile(r'(?im)\?=')
  INDENT_REGEX = re.compile(r'\n\s+')
  ADDRESS_HEADERS_WHITELIST = ['From', 'To', 'Delivered-To', 'Cc...
  __package__ = 'lamson'
Function Details [hide private]

VALUE_IS_EMAIL_ADDRESS(v)

source code 

@

from_message(message)

source code 

Given a MIMEBase or similar Python email API message object, this will canonicalize it and give you back a pristine MailBase. If it can't then it raises a EncodingError.

to_message(mail)

source code 

Given a MailBase message, this will construct a MIMEPart that is canonicalized for use with the Python email API.

to_string(mail, envelope_header=False)

source code 

Returns a canonicalized email string you can use to send or store somewhere.

from_string(data)

source code 

Takes a string, and tries to clean it up into a clean MailBase.

to_file(mail, fileobj)

source code 

Writes a canonicalized message to the given file.

from_file(fileobj)

source code 

Reads an email and cleans it up to make a MailBase.

normalize_header(header)

source code 

parse_parameter_header(message, header)

source code 

decode_message_body(mail, message)

source code 

properly_encode_header(value, encoder, not_email)

source code 

The only thing special (weird) about this function is that it tries to do a fast check to see if the header value has an email address in it. Since random headers could have an email address, and email addresses have weird special formatting rules, we have to check for it.

Normally this works fine, but in Librelist, we need to "obfuscate" email addresses by changing the '@' to '-AT-'. This is where VALUE_IS_EMAIL_ADDRESS exists. It's a simple lambda returning True/False to check if a header value has an email address. If you need to make this check different, then change this.

header_to_mime_encoding(value, not_email=False)

source code 

header_from_mime_encoding(header)

source code 

guess_encoding_and_decode(original, data, errors='strict')

source code 

attempt_decoding(charset, dec)

source code 

apply_charset_to_header(charset, encoding, data)

source code 

_match(data, pattern, pos)

source code 

_tokenize(data, next)

source code 

_scan(data)

source code 

_parse_charset_header(data)

source code 

properly_decode_header(header)

source code 

Variables Details [hide private]

DEFAULT_ENCODING

Value:
'utf-8'

DEFAULT_ERROR_HANDLING

Value:
'strict'

CONTENT_ENCODING_KEYS

Value:
set(['Content-Disposition',
     'Content-Transfer-Encoding',
     'Content-Type',
     'Mime-Version'])

CONTENT_ENCODING_REMOVED_PARAMS

Value:
['boundary']

REGEX_OPTS

Value:
10

ENCODING_REGEX

Value:
re.compile(r'(?im)=\?([a-z0-9-]+?)\?([bq])\?')

ENCODING_END_REGEX

Value:
re.compile(r'(?im)\?=')

INDENT_REGEX

Value:
re.compile(r'\n\s+')

ADDRESS_HEADERS_WHITELIST

Value:
['From', 'To', 'Delivered-To', 'Cc', 'Bcc']

__package__

Value:
'lamson'