Merged revisions 67952-67953,67955,67957-67958,67960-67961,67963,67965,67967,67970-67971,67973,67982,67988,67990,67995,68014,68016,68030,68057,68061,68112,68115-68118,68120-68121,68123-68128 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r67952 | georg.brandl | 2008-12-27 18:42:40 +0100 (Sat, 27 Dec 2008) | 2 lines

  #4752: actually use custom handler in example.
........
  r67953 | georg.brandl | 2008-12-27 19:20:04 +0100 (Sat, 27 Dec 2008) | 3 lines

  Patch #4739 by David Laban: add symbols to pydoc help topics,
  so that ``help('@')`` works as expected.
........
  r67955 | georg.brandl | 2008-12-27 19:27:53 +0100 (Sat, 27 Dec 2008) | 3 lines

  Follow-up to r67746 in order to restore backwards-compatibility for
  those who (monkey-)patch TextWrapper.wordsep_re with a custom RE.
........
  r67957 | georg.brandl | 2008-12-27 19:49:19 +0100 (Sat, 27 Dec 2008) | 2 lines

  #4754: improve winsound documentation.
........
  r67958 | georg.brandl | 2008-12-27 20:02:59 +0100 (Sat, 27 Dec 2008) | 2 lines

  #4682: 'b' is actually unsigned char.
........
  r67960 | georg.brandl | 2008-12-27 20:04:44 +0100 (Sat, 27 Dec 2008) | 2 lines

  #4695: fix backslashery.
........
  r67961 | georg.brandl | 2008-12-27 20:06:04 +0100 (Sat, 27 Dec 2008) | 2 lines

  Use :samp: role.
........
  r67963 | georg.brandl | 2008-12-27 20:11:15 +0100 (Sat, 27 Dec 2008) | 2 lines

  #4671: document that pydoc imports modules.
........
  r67965 | antoine.pitrou | 2008-12-27 21:34:52 +0100 (Sat, 27 Dec 2008) | 3 lines

  Issue #4677: add two list comprehension tests to pybench.
........
  r67967 | benjamin.peterson | 2008-12-27 23:18:58 +0100 (Sat, 27 Dec 2008) | 1 line

  fix markup
........
  r67970 | alexandre.vassalotti | 2008-12-28 02:52:58 +0100 (Sun, 28 Dec 2008) | 2 lines

  Fix name mangling of PyUnicode_ClearFreeList.
........
  r67971 | alexandre.vassalotti | 2008-12-28 03:10:35 +0100 (Sun, 28 Dec 2008) | 2 lines

  Sort UCS-2/UCS-4 name mangling list.
........
  r67973 | alexandre.vassalotti | 2008-12-28 03:58:22 +0100 (Sun, 28 Dec 2008) | 2 lines

  Document Py_VaBuildValue.
........
  r67982 | benjamin.peterson | 2008-12-28 16:37:31 +0100 (Sun, 28 Dec 2008) | 1 line

  fix WORD_BIGEDIAN declaration in Universal builds; fixes #4060 and #4728
........
  r67988 | ronald.oussoren | 2008-12-28 20:40:56 +0100 (Sun, 28 Dec 2008) | 1 line

  Issue4064: architecture string for universal builds on OSX
........
  r67990 | ronald.oussoren | 2008-12-28 20:50:40 +0100 (Sun, 28 Dec 2008) | 3 lines

  Update the fix for issue4064 to deal correctly with all three variants of
  universal builds that are presented by the configure script.
........
  r67995 | benjamin.peterson | 2008-12-28 22:16:07 +0100 (Sun, 28 Dec 2008) | 1 line

  #4763 PyErr_ExceptionMatches won't blow up with NULL arguments
........
  r68014 | benjamin.peterson | 2008-12-29 18:47:42 +0100 (Mon, 29 Dec 2008) | 1 line

  #4764 set IOError.filename when trying to open a directory on POSIX platforms
........
  r68016 | benjamin.peterson | 2008-12-29 18:56:58 +0100 (Mon, 29 Dec 2008) | 1 line

  #4764 in io.open, set IOError.filename when trying to open a directory on POSIX platforms
........
  r68030 | benjamin.peterson | 2008-12-29 22:38:14 +0100 (Mon, 29 Dec 2008) | 1 line

  fix French
........
  r68057 | vinay.sajip | 2008-12-30 08:01:25 +0100 (Tue, 30 Dec 2008) | 1 line

  Minor documentation change relating to NullHandler.
........
  r68061 | georg.brandl | 2008-12-30 11:15:49 +0100 (Tue, 30 Dec 2008) | 2 lines

  #4778: attributes can't be called.
........
  r68112 | benjamin.peterson | 2009-01-01 00:48:39 +0100 (Thu, 01 Jan 2009) | 1 line

  #4795 inspect.isgeneratorfunction() should return False instead of None
........
  r68115 | benjamin.peterson | 2009-01-01 05:04:41 +0100 (Thu, 01 Jan 2009) | 1 line

  simplfy code
........
  r68116 | georg.brandl | 2009-01-01 12:46:51 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4100: note that element children are not necessarily present on "start" events.
........
  r68117 | georg.brandl | 2009-01-01 12:53:55 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4156: make clear that "protocol" is to be replaced with the protocol name.
........
  r68118 | georg.brandl | 2009-01-01 13:00:19 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4185: clarify escape behavior of replacement strings.
........
  r68120 | georg.brandl | 2009-01-01 13:15:31 +0100 (Thu, 01 Jan 2009) | 4 lines

  #4228: Pack negative values the same way as 2.4
  in struct's L format.
........
  r68121 | georg.brandl | 2009-01-01 13:43:33 +0100 (Thu, 01 Jan 2009) | 2 lines

  Point to types module in new module deprecation notice.
........
  r68123 | georg.brandl | 2009-01-01 13:52:29 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4784: ... on three counts ...
........
  r68124 | georg.brandl | 2009-01-01 13:53:19 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4782: Fix markup error that hid load() and loads().
........
  r68125 | georg.brandl | 2009-01-01 14:02:09 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4776: add data_files and package_dir arguments.
........
  r68126 | georg.brandl | 2009-01-01 14:05:13 +0100 (Thu, 01 Jan 2009) | 2 lines

  Handlers are in the `logging.handlers` module.
........
  r68127 | georg.brandl | 2009-01-01 14:14:49 +0100 (Thu, 01 Jan 2009) | 2 lines

  #4767: Use correct submodules for all MIME classes.
........
  r68128 | antoine.pitrou | 2009-01-01 15:11:22 +0100 (Thu, 01 Jan 2009) | 3 lines

  Issue #3680: Reference cycles created through a dict, set or deque iterator did not get collected.
........

11 files changed, 155 insertions, 20 deletions

diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 48cc17f624..e9d29ff49c 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -140,9 +140,13 @@ def get_platform ():
                 # 'universal' instead of 'fat'.
 
                 machine = 'fat'
+                cflags = get_config_vars().get('CFLAGS')
 
-                if '-arch x86_64' in get_config_vars().get('CFLAGS'):
-                    machine = 'universal'
+                if '-arch x86_64' in cflags:
+                    if '-arch i386' in cflags:
+                        machine = 'universal'
+                    else:
+                        machine = 'fat64'
 
             elif machine in ('PowerPC', 'Power_Macintosh'):
                 # Pick a sane name for the PPC architecture.
diff --git a/Lib/inspect.py b/Lib/inspect.py
index d3d946d525..1685bfc98d 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -158,9 +158,8 @@ def isgeneratorfunction(object):
     Generator function objects provides same attributes as functions.
 
     See isfunction.__doc__ for attributes listing."""
-    if (isfunction(object) or ismethod(object)) and \
-        object.func_code.co_flags & CO_GENERATOR:
-        return True
+    return bool((isfunction(object) or ismethod(object)) and
+                object.func_code.co_flags & CO_GENERATOR)
 
 def isgenerator(object):
     """Return true if the object is a generator.
diff --git a/Lib/pydoc.py b/Lib/pydoc.py
index 179026b051..dda0208edf 100755
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1574,6 +1574,42 @@ class Helper:
         'with': ('with', 'CONTEXTMANAGERS EXCEPTIONS yield'),
         'yield': ('yield', ''),
     }
+    # Either add symbols to this dictionary or to the symbols dictionary
+    # directly: Whichever is easier. They are merged later.
+    _symbols_inverse = {
+        'STRINGS' : ("'", "'''", "r'", "u'", '"""', '"', 'r"', 'u"'),
+        'OPERATORS' : ('+', '-', '*', '**', '/', '//', '%', '<<', '>>', '&',
+                       '|', '^', '~', '<', '>', '<=', '>=', '==', '!=', '<>'),
+        'COMPARISON' : ('<', '>', '<=', '>=', '==', '!=', '<>'),
+        'UNARY' : ('-', '~'),
+        'AUGMENTEDASSIGNMENT' : ('+=', '-=', '*=', '/=', '%=', '&=', '|=',
+                                '^=', '<<=', '>>=', '**=', '//='),
+        'BITWISE' : ('<<', '>>', '&', '|', '^', '~'),
+        'COMPLEX' : ('j', 'J')
+    }
+    symbols = {
+        '%': 'OPERATORS FORMATTING',
+        '**': 'POWER',
+        ',': 'TUPLES LISTS FUNCTIONS',
+        '.': 'ATTRIBUTES FLOAT MODULES OBJECTS',
+        '...': 'ELLIPSIS',
+        ':': 'SLICINGS DICTIONARYLITERALS',
+        '@': 'def class',
+        '\\': 'STRINGS',
+        '_': 'PRIVATENAMES',
+        '__': 'PRIVATENAMES SPECIALMETHODS',
+        '`': 'BACKQUOTES',
+        '(': 'TUPLES FUNCTIONS CALLS',
+        ')': 'TUPLES FUNCTIONS CALLS',
+        '[': 'LISTS SUBSCRIPTS SLICINGS',
+        ']': 'LISTS SUBSCRIPTS SLICINGS'
+    }
+    for topic, symbols_ in _symbols_inverse.iteritems():
+        for symbol in symbols_:
+            topics = symbols.get(symbol, topic)
+            if topic not in topics:
+                topics = topics + ' ' + topic
+            symbols[symbol] = topics
 
     topics = {
         'TYPES': ('types', 'STRINGS UNICODE NUMBERS SEQUENCES MAPPINGS '
@@ -1717,10 +1753,12 @@ has the same effect as typing a particular string at the help> prompt.
         if type(request) is type(''):
             if request == 'help': self.intro()
             elif request == 'keywords': self.listkeywords()
+            elif request == 'symbols': self.listsymbols()
             elif request == 'topics': self.listtopics()
             elif request == 'modules': self.listmodules()
             elif request[:8] == 'modules ':
                 self.listmodules(split(request)[1])
+            elif request in self.symbols: self.showsymbol(request)
             elif request in self.keywords: self.showtopic(request)
             elif request in self.topics: self.showtopic(request)
             elif request: doc(request, 'Help on %s:')
@@ -1766,6 +1804,14 @@ Here is a list of the Python keywords.  Enter any keyword to get more help.
 ''')
         self.list(self.keywords.keys())
 
+    def listsymbols(self):
+        self.output.write('''
+Here is a list of the punctuation symbols which Python assigns special meaning
+to. Enter any symbol to get more help.
+
+''')
+        self.list(self.symbols.keys())
+
     def listtopics(self):
         self.output.write('''
 Here is a list of available topics.  Enter any topic name to get more help.
@@ -1773,7 +1819,7 @@ Here is a list of available topics.  Enter any topic name to get more help.
 ''')
         self.list(self.topics.keys())
 
-    def showtopic(self, topic):
+    def showtopic(self, topic, more_xrefs=''):
         try:
             import pydoc_topics
         except ImportError:
@@ -1787,7 +1833,7 @@ module "pydoc_topics" could not be found.
             self.output.write('no documentation found for %s\n' % repr(topic))
             return
         if type(target) is type(''):
-            return self.showtopic(target)
+            return self.showtopic(target, more_xrefs)
 
         label, xrefs = target
         try:
@@ -1796,6 +1842,8 @@ module "pydoc_topics" could not be found.
             self.output.write('no documentation found for %s\n' % repr(topic))
             return
         pager(strip(doc) + '\n')
+        if more_xrefs:
+            xrefs = (xrefs or '') + ' ' + more_xrefs
         if xrefs:
             import StringIO, formatter
             buffer = StringIO.StringIO()
@@ -1803,6 +1851,11 @@ module "pydoc_topics" could not be found.
                 'Related help topics: ' + join(split(xrefs), ', ') + '\n')
             self.output.write('\n%s\n' % buffer.getvalue())
 
+    def showsymbol(self, symbol):
+        target = self.symbols[symbol]
+        topic, _, xrefs = target.partition(' ')
+        self.showtopic(topic, xrefs)
+
     def listmodules(self, key=''):
         if key:
             self.output.write('''
diff --git a/Lib/re.py b/Lib/re.py
index ee962bbc8f..2cf7132d3d 100644
--- a/Lib/re.py
+++ b/Lib/re.py
@@ -145,7 +145,8 @@ def sub(pattern, repl, string, count=0):
     """Return the string obtained by replacing the leftmost
     non-overlapping occurrences of the pattern in string by the
     replacement repl.  repl can be either a string or a callable;
-    if a callable, it's passed the match object and must return
+    if a string, backslash escapes in it are processed.  If it is
+    a callable, it's passed the match object and must return
     a replacement string to be used."""
     return _compile(pattern, 0).sub(repl, string, count)
 
@@ -155,7 +156,8 @@ def subn(pattern, repl, string, count=0):
     non-overlapping occurrences of the pattern in the source
     string by the replacement repl.  number is the number of
     substitutions that were made. repl can be either a string or a
-    callable; if a callable, it's passed the match object and must
+    callable; if a string, backslash escapes in it are processed.
+    If it is a callable, it's passed the match object and must
     return a replacement string to be used."""
     return _compile(pattern, 0).subn(repl, string, count)
 
diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py
index 0f0d09847e..4e2de3d005 100644
--- a/Lib/test/test_deque.py
+++ b/Lib/test/test_deque.py
@@ -1,7 +1,8 @@
 from collections import deque
 import unittest
 from test import test_support, seq_tests
-from weakref import proxy
+import gc
+import weakref
 import copy
 import cPickle as pickle
 import random
@@ -418,6 +419,22 @@ class TestBasic(unittest.TestCase):
             d.append(1)
             gc.collect()
 
+    def test_container_iterator(self):
+        # Bug # XXX: tp_traverse was not implemented for deque iterator objects
+        class C(object):
+            pass
+        for i in range(2):
+            obj = C()
+            ref = weakref.ref(obj)
+            if i == 0:
+                container = deque([obj, 1])
+            else:
+                container = reversed(deque([obj, 1]))
+            obj.x = iter(container)
+            del obj, container
+            gc.collect()
+            self.assert_(ref() is None, "Cycle was not collected")
+
 class TestVariousIteratorArgs(unittest.TestCase):
 
     def test_constructor(self):
@@ -528,7 +545,7 @@ class TestSubclass(unittest.TestCase):
 
     def test_weakref(self):
         d = deque('gallahad')
-        p = proxy(d)
+        p = weakref.proxy(d)
         self.assertEqual(str(p), str(d))
         d = None
         self.assertRaises(ReferenceError, str, p)
diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
index f715657d36..14d62f5024 100644
--- a/Lib/test/test_dict.py
+++ b/Lib/test/test_dict.py
@@ -2,6 +2,7 @@ import unittest
 from test import test_support
 
 import UserDict, random, string
+import gc, weakref
 
 
 class DictTest(unittest.TestCase):
@@ -554,6 +555,19 @@ class DictTest(unittest.TestCase):
             pass
         d = {}
 
+    def test_container_iterator(self):
+        # Bug # XXX: tp_traverse was not implemented for dictiter objects
+        class C(object):
+            pass
+        iterators = (dict.iteritems, dict.itervalues, dict.iterkeys)
+        for i in iterators:
+            obj = C()
+            ref = weakref.ref(obj)
+            container = {obj: 1}
+            obj.x = i(container)
+            del obj, container
+            gc.collect()
+            self.assert_(ref() is None, "Cycle was not collected")
 
 
 from test import mapping_tests
diff --git a/Lib/test/test_file.py b/Lib/test/test_file.py
index 96f6da2023..b4f494ba96 100644
--- a/Lib/test/test_file.py
+++ b/Lib/test/test_file.py
@@ -125,6 +125,19 @@ class AutoFileTests(unittest.TestCase):
 
 class OtherFileTests(unittest.TestCase):
 
+    def testOpenDir(self):
+        this_dir = os.path.dirname(__file__)
+        for mode in (None, "w"):
+            try:
+                if mode:
+                    f = open(this_dir, mode)
+                else:
+                    f = open(this_dir)
+            except IOError as e:
+                self.assertEqual(e.filename, this_dir)
+            else:
+                self.fail("opening a directory didn't raise an IOError")
+
     def testModeStrings(self):
         # check invalid mode strings
         for mode in ("", "aU", "wU+"):
diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py
index c9787795e3..d8cf415ed8 100644
--- a/Lib/test/test_fileio.py
+++ b/Lib/test/test_fileio.py
@@ -109,6 +109,7 @@ class AutoFileTests(unittest.TestCase):
             _fileio._FileIO('.', 'r')
         except IOError as e:
             self.assertNotEqual(e.errno, 0)
+            self.assertEqual(e.filename, ".")
         else:
             self.fail("Should have raised IOError")
 
diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py
index d38a675907..8d05712365 100644
--- a/Lib/test/test_set.py
+++ b/Lib/test/test_set.py
@@ -1,6 +1,7 @@
 import unittest
 from test import test_support
-from weakref import proxy
+import gc
+import weakref
 import operator
 import copy
 import pickle
@@ -322,6 +323,18 @@ class TestJointOps(unittest.TestCase):
         self.assertEqual(sum(elem.hash_count for elem in d), n)
         self.assertEqual(d3, dict.fromkeys(d, 123))
 
+    def test_container_iterator(self):
+        # Bug # XXX: tp_traverse was not implemented for set iterator object
+        class C(object):
+            pass
+        obj = C()
+        ref = weakref.ref(obj)
+        container = set([obj, 1])
+        obj.x = iter(container)
+        del obj, container
+        gc.collect()
+        self.assert_(ref() is None, "Cycle was not collected")
+
 class TestSet(TestJointOps):
     thetype = set
 
@@ -538,7 +551,7 @@ class TestSet(TestJointOps):
 
     def test_weakref(self):
         s = self.thetype('gallahad')
-        p = proxy(s)
+        p = weakref.proxy(s)
         self.assertEqual(str(p), str(s))
         s = None
         self.assertRaises(ReferenceError, str, p)
diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py
index 232bffc275..7f5f08b4c6 100644
--- a/Lib/test/test_struct.py
+++ b/Lib/test/test_struct.py
@@ -2,6 +2,8 @@ import array
 import unittest
 import struct
 import warnings
+warnings.filterwarnings("ignore", "struct integer overflow masking is deprecated",
+                        DeprecationWarning)
 
 from functools import wraps
 from test.test_support import TestFailed, verbose, run_unittest
@@ -461,6 +463,11 @@ class StructTest(unittest.TestCase):
                 self.check_float_coerce(endian + fmt, 1.0)
                 self.check_float_coerce(endian + fmt, 1.5)
 
+    def test_issue4228(self):
+        # Packing a long may yield either 32 or 64 bits
+        x = struct.pack('L', -1)[:4]
+        self.assertEqual(x, '\xff'*4)
+
     def test_unpack_from(self):
         test_string = 'abcd01234'
         fmt = '4s'
diff --git a/Lib/textwrap.py b/Lib/textwrap.py
index 192b43b1df..64a5b97c92 100644
--- a/Lib/textwrap.py
+++ b/Lib/textwrap.py
@@ -84,7 +84,7 @@ class TextWrapper:
     # splits into
     #   Hello/ /there/ /--/ /you/ /goof-/ball,/ /use/ /the/ /-b/ /option!
     # (after stripping out empty strings).
-    wordsep_re = (
+    wordsep_re = re.compile(
         r'(\s+|'                                  # any whitespace
         r'[^\s\w]*\w+[^0-9\W]-(?=\w+[^0-9\W])|'   # hyphenated words
         r'(?<=[\w\!\"\'\&\.\,\?])-{2,}(?=\w))')   # em-dash
@@ -93,7 +93,7 @@ class TextWrapper:
     #   "Hello there -- you goof-ball, use the -b option!"
     # splits into
     #   Hello/ /there/ /--/ /you/ /goof-ball,/ /use/ /the/ /-b/ /option!/
-    wordsep_simple_re = r'(\s+)'
+    wordsep_simple_re = re.compile(r'(\s+)')
 
     # XXX this is not locale- or charset-aware -- string.lowercase
     # is US-ASCII only (and therefore English-only)
@@ -124,6 +124,13 @@ class TextWrapper:
         self.drop_whitespace = drop_whitespace
         self.break_on_hyphens = break_on_hyphens
 
+        # recompile the regexes for Unicode mode -- done in this clumsy way for
+        # backwards compatibility because it's rather common to monkey-patch
+        # the TextWrapper class' wordsep_re attribute.
+        self.wordsep_re_uni = re.compile(self.wordsep_re.pattern, re.U)
+        self.wordsep_simple_re_uni = re.compile(
+            self.wordsep_simple_re.pattern, re.U)
+
 
     # -- Private methods -----------------------------------------------
     # (possibly useful for subclasses to override)
@@ -160,12 +167,17 @@ class TextWrapper:
           'use', ' ', 'the', ' ', '-b', ' ', option!'
         otherwise.
         """
-        flags = re.UNICODE if isinstance(text, unicode) else 0
-        if self.break_on_hyphens:
-            pat = self.wordsep_re
+        if isinstance(text, unicode):
+            if self.break_on_hyphens:
+                pat = self.wordsep_re_uni
+            else:
+                pat = self.wordsep_simple_re_uni
         else:
-            pat = self.wordsep_simple_re
-        chunks = re.compile(pat, flags).split(text)
+            if self.break_on_hyphens:
+                pat = self.wordsep_re
+            else:
+                pat = self.wordsep_simple_re
+        chunks = pat.split(text)
         chunks = filter(None, chunks)  # remove empty chunks
         return chunks