Implement --editable option, which allows you to just download and extract

(or check out from Subversion) one or more source distributions, without
actually building or installing them (or their dependencies).

--HG--
branch : setuptools
extra : convert_revision : svn%3A6015fed2-1504-0410-9fe1-9d1591cc4771/sandbox/trunk/setuptools%4041147

4 files changed, 225 insertions, 151 deletions

diff --git a/EasyInstall.txt b/EasyInstall.txt
index d48757cc..a5605fbe 100755
--- a/EasyInstall.txt
+++ b/EasyInstall.txt
@@ -88,6 +88,17 @@ extracted in the current directory (New in 0.5a9)::
 
     easy_install .
 
+**Example 7**.  (New in 0.6a1) Find a source distribution or Subversion
+checkout URL for a package, and extract it or check it out to
+``~/projects/sqlobject`` (the name will always be in all-lowercase), where it
+can be examined or edited.  (The package will not be installed, but it can
+easily be installed with ``easy_install ~/projects/sqlobject``.  See `Editing
+and Viewing Source Packages`_ below for more info.)::
+
+    easy_install --editable --build-directory ~/projects SQLObject
+
+Note:
+
 Easy Install accepts URLs, filenames, PyPI package names (i.e., ``distutils``
 "distribution" names), and package+version specifiers.  In each case, it will
 attempt to locate the latest available version that meets your criteria.
@@ -270,6 +281,44 @@ below for a list of the standard configuration file locations, and links to
 more documentation on using distutils configuration files.
 
 
+Editing and Viewing Source Packages
+-----------------------------------
+
+Sometimes a package's source distribution  contains additional documentation,
+examples, configuration files, etc., that are not part of its actual code.  If
+you want to be able to examine these files, you can use the ``--editable``
+option to EasyInstall, and EasyInstall will look for a source distribution
+or Subversion URL for the package, then download and extract it or check it out
+as a subdirectory of the ``--build-directory`` you specify.  If you then wish
+to install the package after editing or configuring it, you can do so by
+rerunning EasyInstall with that directory as the target.
+
+Note that using ``--editable`` stops EasyInstall from actually building or
+installing the package; it just finds, obtains, and possibly unpacks it for
+you.  This allows you to make changes to the package if necessary, and to
+either install it in development mode using ``setup.py develop`` (if the
+package uses setuptools, that is), or by running ``easy_install projectdir``
+(where ``projectdir`` is the subdirectory EasyInstall created for the
+downloaded package.
+
+In order to use ``--editable`` (``-e`` for short), you *must* also supply a
+``--build-directory`` (``-b`` for short).  The project will be placed in a
+subdirectory of the build directory.  The subdirectory will have the same
+name as the project itself, but in all-lowercase.  If a file or directory of
+that name already exists, EasyInstall will print an error message and exit.
+
+Also, when using ``--editable``, you cannot use URLs or filenames as arguments.
+You *must* specify project names (and optional version requirements) so that
+EasyInstall knows what directory name(s) to create.  If you need to force
+EasyInstall to use a particular URL or filename, you should specify it as a
+``--find-links`` item (``-f`` for short), and then also specify
+the project name, e.g.::
+
+    easy_install -eb ~/projects \
+     -fhttp://prdownloads.sourceforge.net/ctypes/ctypes-0.9.6.tar.gz?download \
+     ctypes==0.9.6
+
+
 Dealing with Installation Conflicts
 -----------------------------------
 
@@ -530,16 +579,26 @@ Command-Line Options
     package, and those download pages will be searched for links to download
     an egg or source distribution.
 
-``--build-directory=DIR, -b DIR`` (New in 0.3a3)
-    Set the directory used to download, extract, and install the package.  The
-    directory is not cleared before or after installation, so the downloaded
-    packages and extracted contents will remain there afterwards, allowing you
-    to read any documentation, examples, scripts, etc. that may have been
-    included with the source distribution (if any).
-
-    This option can only be used when you are specifying a single installation
-    URL or filename, so that the installer will not be confused by the presence
-    of multiple ``setup.py`` files in the build directory.
+``--editable, -e`` (New in 0.6a1)
+    Only find and download source distributions for the specified projects,
+    unpacking them to subdirectories of the specified ``--build-directory``.
+    EasyInstall will not actually build or install the requested projects or
+    their dependencies; it will just find and extract them for you.  See
+    `Editing and Viewing Source Packages`_ above for more details.
+    
+``--build-directory=DIR, -b DIR`` (UPDATED in 0.6a1)
+    Set the directory used to build source packages.  If a package is built
+    from a source distribution or checkout, it will be extracted to a
+    subdirectory of the specified directory.  The subdirectory will have the
+    same name as the extracted distribution's project, but in all-lowercase.
+    If a file or directory of that name already exists in the given directory,
+    a warning will be printed to the console, and the build will take place in
+    a temporary directory instead.
+
+    This option is most useful in combination with the ``--editable`` option,
+    which forces EasyInstall to *only* find and extract (but not build and
+    install) source distributions.  See `Editing and Viewing Source Packages`_,
+    above, for more information.
 
 ``--verbose, -v, --quiet, -q`` (New in 0.4a4)
     Control the level of detail of EasyInstall's progress messages.  The
@@ -649,6 +708,13 @@ Known Issues
    i.e., all non-alphanumeric runs must be condensed to single underscore
    characters.
 
+ * Added the ``--editable`` option; see `Editing and Viewing Source Packages`_
+   above for more info.  Also, slightly changed the behavior of the
+   ``--build-directory`` option.
+
+ * Fixed the setup script sandbox facility not recognizing certain paths as
+   valid on case-insensitive platforms.
+
 0.5a12
  * Fix ``python -m easy_install`` not working due to setuptools being installed
    as a zipfile.  Update safety scanner to check for modules that might be used
@@ -689,7 +755,7 @@ Known Issues
  * Added the ``--always-unzip/-Z`` option, to force unzipping of packages that
    would ordinarily be considered safe to unzip, and changed the meaning of
    ``--zip-ok/-z`` to "always leave everything zipped".
-   
+
 0.5a8
  * There is now a separate documentation page for `setuptools`_; revision
    history that's not specific to EasyInstall has been moved to that page.
diff --git a/setuptools.txt b/setuptools.txt
index bc6747c8..d00e6d92 100755
--- a/setuptools.txt
+++ b/setuptools.txt
@@ -783,6 +783,14 @@ downloading unless it has a version included in the ``#egg=`` suffix, and
 it's a higher version than EasyInstall has seen in any other links for your
 project.
 
+As a result, it's a common practice to use mark checkout URLs with a version of
+"dev" (i.e., ``#egg=projectname-dev``), so that users can do something like
+this::
+
+    easy_install --editable projectname==dev
+
+in order to check out the in-development version of ``projectname``.
+
 
 Distributing Extensions compiled with Pyrex
 -------------------------------------------
diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
index 37aa496a..110ad211 100755
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -69,17 +69,17 @@ class easy_install(Command):
          "filename in which to record list of installed files"),
         ('always-unzip', 'Z', "don't install as a zipfile, no matter what"),
         ('site-dirs=','S',"list of directories where .pth files work"),
+        ('editable', 'e', "Install specified packages in editable form"),
     ]
 
     boolean_options = [
         'zip-ok', 'multi-version', 'exclude-scripts', 'upgrade', 'always-copy',
-        'delete-conflicting', 'ignore-conflicts-at-my-risk',
+        'delete-conflicting', 'ignore-conflicts-at-my-risk', 'editable',
     ]
 
     negative_opt = {'always-unzip': 'zip-ok'}
     create_index = PackageIndex
 
-
     def initialize_options(self):
         self.zip_ok = None
         self.install_dir = self.script_dir = self.exclude_scripts = None
@@ -89,6 +89,8 @@ class easy_install(Command):
         self.args = None
         self.optimize = self.record = None
         self.upgrade = self.always_copy = self.multi_version = None
+        self.editable = None
+
         # Options not specifiable via command line
         self.package_index = None
         self.pth_file = None
@@ -119,8 +121,6 @@ class easy_install(Command):
 
 
 
-
-
     def finalize_options(self):
         # If a non-default installation directory was specified, default the
         # script directory to match it.
@@ -203,27 +203,18 @@ class easy_install(Command):
                 "--ignore-conflicts-at-my-risk at the same time"
             )
 
-        if not self.args:
+        if self.editable and not self.build_directory:
             raise DistutilsArgError(
-                "No urls, filenames, or requirements specified (see --help)")
+                "Must specify a build directory (-b) when using --editable"
+            )
 
-        elif len(self.args)>1 and self.build_directory is not None:
+        if not self.args:
             raise DistutilsArgError(
-                "Build directory can only be set when using one URL"
-            )
+                "No urls, filenames, or requirements specified (see --help)")
 
         self.outputs = []
 
 
-    def alloc_tmp(self):
-        if self.build_directory is None:
-            return tempfile.mkdtemp(prefix="easy_install-")
-        tmpdir = os.path.realpath(self.build_directory)
-        if not os.path.isdir(tmpdir):
-            os.makedirs(tmpdir)
-        return tmpdir
-
-
     def run(self):
         if self.verbose<>self.distribution.verbose:
             log.set_verbosity(self.verbose)
@@ -244,21 +235,6 @@ class easy_install(Command):
 
 
 
-    def add_output(self, path):
-        if os.path.isdir(path):
-            for base, dirs, files in os.walk(path):
-                for filename in files:
-                    self.outputs.append(os.path.join(base,filename))
-        else:
-            self.outputs.append(path)
-
-
-
-
-
-
-
-
 
 
 
@@ -268,9 +244,32 @@ class easy_install(Command):
 
 
 
+    def add_output(self, path):
+        if os.path.isdir(path):
+            for base, dirs, files in os.walk(path):
+                for filename in files:
+                    self.outputs.append(os.path.join(base,filename))
+        else:
+            self.outputs.append(path)
 
+    def not_editable(self, spec):
+        if self.editable:
+            raise DistutilsArgError(
+                "Invalid argument %r: you can't use filenames or URLs "
+                "with --editable (except via the --find-links option)."
+                % (spec,)
+            )
 
+    def check_editable(self,spec):
+        if not self.editable:
+            return
 
+        if os.path.exists(os.path.join(self.build_directory, spec.key)):
+            raise DistutilsArgError(
+                "%r already exists in %s; can't do a checkout there" %
+                (spec.key, self.build_directory)
+            )
+            
 
 
 
@@ -282,36 +281,33 @@ class easy_install(Command):
 
 
 
+        
 
 
 
     def easy_install(self, spec):
-        tmpdir = self.alloc_tmp()
+        tmpdir = tempfile.mkdtemp(prefix="easy_install-")
         download = None
 
         try:
             if not isinstance(spec,Requirement):
                 if URL_SCHEME(spec):
                     # It's a url, download it to tmpdir and process
+                    self.not_editable(spec)
                     download = self.package_index.download(spec, tmpdir)
                     return self.install_item(None, download, tmpdir, True)
 
                 elif os.path.exists(spec):
                     # Existing file or directory, just process it directly
+                    self.not_editable(spec)
                     return self.install_item(None, spec, tmpdir, True)
                 else:
-                    try:
-                        spec = Requirement.parse(spec)
-                    except ValueError:
-                        raise DistutilsError(
-                            "Not a URL, existing file, or requirement spec: %r"
-                            % (spec,)
-                        )
-
-            if isinstance(spec, Requirement):
-                download = self.package_index.fetch(spec, tmpdir, self.upgrade)
-            else:
-                spec = None
+                    spec = parse_requirement_arg(spec)
+
+            self.check_editable(spec)
+            download = self.package_index.fetch(
+                spec, tmpdir, self.upgrade, self.editable
+            )
 
             if download is None:
                 raise DistutilsError(
@@ -321,18 +317,22 @@ class easy_install(Command):
             return self.install_item(spec, download, tmpdir)
 
         finally:
-            if self.build_directory is None:
+            if os.path.exists(tmpdir):
                 shutil.rmtree(tmpdir)
 
 
 
+
+
+
+
     def install_item(self, spec, download, tmpdir, install_needed=False):
         # Installation is also needed if file in tmpdir or is not an egg
         install_needed = install_needed or os.path.dirname(download) == tmpdir
         install_needed = install_needed or not download.endswith('.egg')
         log.info("Processing %s", os.path.basename(download))
         if install_needed or self.always_copy:
-            dists = self.install_eggs(download, tmpdir)
+            dists = self.install_eggs(spec, download, tmpdir)
             for dist in dists:
                 self.process_distribution(spec, dist)
         else:
@@ -386,28 +386,28 @@ class easy_install(Command):
             return True
         return False
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+    def maybe_move(self, spec, dist_filename, setup_base):
+        dst = os.path.join(self.build_directory, spec.key)
+        if os.path.exists(dst):
+            log.warn(
+               "%r already exists in %s; build directory %s will not be kept",
+               spec.key, self.build_directory, setup_base
+            )
+            return setup_base
+        if os.path.isdir(dist_filename):
+            setup_base = dist_filename
+        else:
+            if os.path.dirname(dist_filename)==setup_base:
+                os.unlink(dist_filename)   # get it out of the tmp dir
+            contents = os.listdir(setup_base)
+            if len(contents)==1:
+                dist_filename = os.path.join(setup_base,contents[0])
+                if os.path.isdir(dist_filename):
+                    # if the only thing there is a directory, move it instead
+                    setup_base = dist_filename
+        ensure_directory(dst); shutil.move(setup_base, dst)
+        return dst
+        
     def install_script(self, dist, script_name, script_text, dev_path=None):
         log.info("Installing %s script to %s", script_name,self.script_dir)
         target = os.path.join(self.script_dir, script_name)
@@ -449,7 +449,7 @@ class easy_install(Command):
                 pass
 
 
-    def install_eggs(self, dist_filename, tmpdir):
+    def install_eggs(self, spec, dist_filename, tmpdir):
         # .egg dirs or files are already built, so just return them
         if dist_filename.lower().endswith('.egg'):
             return [self.install_egg(dist_filename, tmpdir)]
@@ -461,8 +461,12 @@ class easy_install(Command):
         if os.path.isfile(dist_filename):
             unpack_archive(dist_filename, tmpdir, self.unpack_progress)
         elif os.path.isdir(dist_filename):
+            # note that setup_base==tmpdir here if this is a svn checkout
             setup_base = os.path.abspath(dist_filename)
 
+        if setup_base==tmpdir and self.build_directory and spec is not None:
+            setup_base = self.maybe_move(spec, dist_filename, setup_base)
+
         # Find the setup.py file
         setup_script = os.path.join(setup_base, 'setup.py')
 
@@ -479,15 +483,11 @@ class easy_install(Command):
             setup_script = setups[0]
 
         # Now run it, and return the result
-        return self.build_and_install(setup_script, setup_base)
-
-
-
-
-
-
-
-
+        if self.editable:
+            log.warn(self.report_editable(spec, setup_script))
+            return []
+        else:
+            return self.build_and_install(setup_script, setup_base)
 
 
     def egg_distribution(self, egg_path):
@@ -723,24 +723,24 @@ PYTHONPATH, or by being added to sys.path by your code.)
         version = dist.version
         return msg % locals()
 
+    def report_editable(self, spec, setup_script):
+        dirname = os.path.dirname(setup_script)
+        python = sys.executable
+        return """\nExtracted editable version of %(spec)s to %(dirname)s
 
+If it uses setuptools in its setup script, you can activate it in
+"development" mode by going to that directory and running::
 
+    %(python)s setup.py --develop
 
+See the setuptools documentation for the "develop" command for more info.
+""" % locals()
 
-
-
-
-
-
-
-
-
-
-    def build_and_install(self, setup_script, setup_base):
+    def run_setup(self, setup_script, setup_base, args):
         sys.modules.setdefault('distutils.command.bdist_egg', bdist_egg)
-        sys.modules.setdefault('distutils.command.bdist_egg', egg_info)
+        sys.modules.setdefault('distutils.command.egg_info', egg_info)
 
-        args = ['bdist_egg', '--dist-dir']
+        args = list(args)
         if self.verbose>2:
             v = 'v' * self.verbose - 1
             args.insert(0,'-'+v)
@@ -748,31 +748,31 @@ PYTHONPATH, or by being added to sys.path by your code.)
             args.insert(0,'-q')
         if self.dry_run:
             args.insert(0,'-n')
+        log.info(
+            "Running %s %s", setup_script[len(setup_base)+1:], ' '.join(args)
+        )
+        try:
+            run_setup(setup_script, args)
+        except SystemExit, v:
+            raise DistutilsError("Setup script exited with %s" % (v.args[0],))
 
-        dist_dir = tempfile.mkdtemp(prefix='egg-dist-tmp-', dir=os.path.dirname(setup_script))
+    def build_and_install(self, setup_script, setup_base):
+        args = ['bdist_egg', '--dist-dir']
+        dist_dir = tempfile.mkdtemp(
+            prefix='egg-dist-tmp-', dir=os.path.dirname(setup_script)
+        )
         try:
             args.append(dist_dir)
-            log.info(
-                "Running %s %s", setup_script[len(setup_base)+1:],
-                ' '.join(args)
-            )
-            try:
-                run_setup(setup_script, args)
-            except SystemExit, v:
-                raise DistutilsError(
-                    "Setup script exited with %s" % (v.args[0],)
-                )
-
+            self.run_setup(setup_script, setup_base, args)
+            all_eggs = AvailableDistributions([dist_dir])
             eggs = []
-            for egg in glob(os.path.join(dist_dir,'*.egg')):
-                eggs.append(self.install_egg(egg, setup_base))
-
+            for key in eggs:
+                for dist in eggs[key]:
+                    eggs.append(self.install_egg(egg, setup_base))
             if not eggs and not self.dry_run:
                 log.warn("No eggs found in %s (setup script problem?)",
                     dist_dir)
-
             return eggs
-
         finally:
             shutil.rmtree(dist_dir)
             log.set_verbosity(self.verbose) # restore our log verbosity
@@ -1010,13 +1010,13 @@ def get_exe_prefixes(exe_filename):
     return prefixes
 
 
-
-
-
-
-
-
-
+def parse_requirement_arg(spec):
+    try:
+        return Requirement.parse(spec)
+    except ValueError:
+        raise DistutilsError(
+            "Not a URL, existing file, or requirement spec: %r" % (spec,)
+        )
 
 
 
diff --git a/setuptools/package_index.py b/setuptools/package_index.py
index 7c4fa49d..d5013795 100755
--- a/setuptools/package_index.py
+++ b/setuptools/package_index.py
@@ -318,7 +318,7 @@ class PackageIndex(AvailableDistributions):
                         (spec,)
                     )
 
-        return self.fetch(spec, tmpdir, force_scan)
+        return self.fetch(spec, tmpdir)
 
 
 
@@ -326,7 +326,7 @@ class PackageIndex(AvailableDistributions):
 
 
 
-    def fetch(self, requirement, tmpdir, force_scan=False):
+    def fetch(self, requirement, tmpdir, force_scan=False, source=False):
         """Obtain a file suitable for fulfilling `requirement`
 
         `requirement` must be a ``pkg_resources.Requirement`` instance.
@@ -336,35 +336,35 @@ class PackageIndex(AvailableDistributions):
         the return value is the same as if you had called the ``download()``
         method with the matching distribution's URL.  If no matching
         distribution is found, returns ``None``.
-        """
 
+        If the `source` flag is set, only source distributions and source
+        checkout links will be considered.
+        """
         # process a Requirement
         self.info("Searching for %s", requirement)
 
+        def find(req):
+            for dist in self.get(req.key, ()):
+                if dist in req and (dist.precedence<=SOURCE_DIST or not source):
+                    self.info("Best match: %s", dist)
+                    return self.download(dist.location, tmpdir)
+            
         if force_scan:
             self.find_packages(requirement)
+            dist = find(requirement)
+        else:
+            dist = find(requirement)
+            if dist is None:
+                self.find_packages(requirement)
+                dist = find(requirement)
 
-        dist = self.best_match(requirement, WorkingSet([]))     # XXX
-
-        if dist is not None:
-            self.info("Best match: %s", dist)
-            return self.download(dist.location, tmpdir)
-
-        self.warn(
-            "No local packages or download links found for %s", requirement
-        )
-        return None
-
-
-
-
-
-
-
-
-
-
-
+        if dist is None:
+            self.warn(
+                "No local packages or download links found for %s%s",
+                (source and "a source distribution of " or ""),
+                requirement,
+            )
+        return dist
 
 
     dl_blocksize = 8192