.. index:: modular crypt format

.. _modular-crypt-format:

.. rst-class:: html-toggle

====================
Modular Crypt Format
====================

.. centered:: *or*, a side note about a standard that isn't

In short, the modular crypt format (MCF) is a standard
for encoding password hash strings, which requires hashes
have the format :samp:`${identifier}${content}`; where
:samp:`{identifier}` is an short alphanumeric string uniquely
identifying a particular scheme, and :samp:`{content}`
is the contents of the scheme, using only the characters
in the regexp range ``[a-zA-Z0-9./]``.

However, there appears to be no central registry of identifiers,
no specification document, and no actual rules;
so the modular crypt format is more of an ad-hoc idea rather than a true standard.

History
=======
Historically, most unix systems supported only :class:`~passlib.hash.des_crypt`.
Around the same time, many incompatible variations were also developed,
but their hashes were not easily distingiushable from each other
(see :ref:`archaic-unix-schemes`); making it impossible to use
multiple hashes on one system, or progressively migrate to a newer scheme.

This was solved with the advent of the MCF,
which was introduced around the time that :class:`~passlib.hash.md5_crypt` was developed.
This format allows hashes from multiple schemes to exist within the same
database, by requiring that all hash strings begin with a unique prefix
using the format :samp:`${identifier}$`.

Requirements
============
Unfortunately, there is no specification document for this format.
Instead, it exists in *de facto* form only; the following
is an attempt to roughly identify the guidelines followed
by the modular crypt format hashes found in passlib:

1. Hash strings must use only 7-bit ascii characters.

   No known OS or application generates hashes which violate this rule.
   However, some systems (eg Linux's shadow routines) will happily
   and correctly accept hashes which contain 8-bit characters in their salt.
   This is probably a case of "permissive in what you accept,
   strict in what you generate".

2. Hash strings should always start with the prefix :samp:`${identifier}$`,
   where :samp:`{identifier}` is a short string uniquely identifying
   hashes generated by that algorithm, using only lower case ascii
   letters, numbers, and hyphens.

   Initially, most schemes adhereing to this format
   only used a single digit to identify the hash
   (eg ``$1$`` for :class:`!md5_crypt`).
   Because of this, many systems only look at the first
   character when attempting to distinguish hashes.

   Despite this, as Unix systems have branched off,
   new hashes have been developed which used larger
   identifying strings (eg ``$sha1$`` for :class:`~passlib.hash.sha1_crypt`);
   so in general identifier strings should not be assumed to use a single character.

3. Hashes should contain only ascii letters ``a``-``z`` and ``A``-``Z``,
   ascii numbers 0-9, and the characters ``./``; though additionally
   they should use the ``$`` character as an internal field separator.

   This is the least adhered-to of any modular crypt format rule.
   Other characters (such as ``=,-``) are sometimes
   used by various formats, though sparingly.

   The only hard and fast stricture
   is that ``:;!*`` and non-printable characters be avoided,
   since this would interfere with parsing of /etc/shadow
   where these hashes are typically stored.

   Pretty much all modular-crypt-format hashes
   use ascii letters, numbers, ``.``, and ``/``
   to provide base64 encoding of their raw data,
   though the exact character value assignments vary between hashes
   (see :data:`passlib.utils.h64`).

4. Hash schemes should put their "checksum" portion
   at the end of the hash, preferrably separated
   by a ``$``.

   This allows password hashes to be easily truncated
   to a "configuration string" containing just
   the identifying prefix, rounds, salt, etc.

   This configuration string then encodes all the information
   generated needed to generate a new hash
   in order to verify a password, without
   having to perform excessive parsing.

   Most modular crypt format hashes follow this,
   though some (like :class:`~passlib.hash.bcrypt`) omit the ``$`` separator.

   As well, there is no set standard about whether configuration
   strings should or should not include a trailing ``$`` at the end,
   though the general rule is that a hash behave the same regardless
   (:class:`~passlib.hash.sun_md5_crypt` behaves particularly poorly
   regarding this last point).

.. note::

    All of the above is guesswork based on examination of existing
    hashes and OS implementations; and was written merely
    to clarify the issue of what the "modular crypt format" is.
    It is drawn from no authoritative sources.

.. index:: modular crypt format; known identifiers

.. _mcf-identifiers:

Identifiers & Platform Support
==============================

The following table lists of all the major MCF hashes supported by Passlib,
and indicates which operating systems [#gae]_ offer native support.

.. todo:: include MacOS X in this list

==================================== ==================== =========== =========== =========== =========== =======
Scheme                               Prefix               Linux       FreeBSD     NetBSD      OpenBSD     Solaris
==================================== ==================== =========== =========== =========== =========== =======
:class:`~passlib.hash.des_crypt`     n/a                  y           y           y           y           y
:class:`~passlib.hash.bsdi_crypt`    ``_``                            y           y           y
:class:`~passlib.hash.md5_crypt`     ``$1$``              y           y           y           y           y
:class:`~passlib.hash.sun_md5_crypt` ``$md5$``, ``$md5,``                                                            y
:class:`~passlib.hash.bcrypt`        ``$2$``, ``$2a$``                y           y           y           y
:class:`~passlib.hash.bsd_nthash`    ``$3$``                          y
:class:`~passlib.hash.sha256_crypt`  ``$5$``              y                                               y
:class:`~passlib.hash.sha512_crypt`  ``$6$``              y                                               y
:class:`~passlib.hash.sha1_crypt`    ``$sha1$``                                   y
==================================== ==================== =========== =========== =========== =========== =======

The following table lists the other MCF hashes supported by Passlib,
most of which are only used by applications:

=========================================== =================== ===========================
Scheme                                      Prefix              Primary Use (if known)
=========================================== =================== ===========================
:class:`~passlib.hash.apr_md5_crypt`        ``$apr1$``          Apache htdigest files
:class:`~passlib.hash.phpass`               ``$P$``, ``$H$``    PHPass-based applications
:class:`~passlib.hash.pbkdf2_sha1`          ``$pbkdf2$``        Passlib-specific
:class:`~passlib.hash.pbkdf2_sha256`        ``$pbkdf2-sha256$`` Passlib-specific
:class:`~passlib.hash.pbkdf2_sha512`        ``$pbkdf2-sha512$`` Passlib-specific
:class:`~passlib.hash.scram`                ``$scram$``         Passlib-specific
:class:`~passlib.hash.cta_pbkdf2_sha1`      ``$p5k2$`` [#cta]_
:class:`~passlib.hash.dlitz_pbkdf2_sha1`    ``$p5k2$`` [#cta]_
=========================================== =================== ===========================

.. rubric:: Footnotes

.. [#gae] As of 2011-08-19, Google App Engine's :mod:`crypt` implementation
          appears to provide hash support matching that of a typical Linux system.

.. [#cta] :class:`!cta_pbkdf2_sha1` and :class:`!dlitz_pbkdf2_sha1` both use
          the same identifier. They can be distinguished
          by the fact that cta hashes will always end in ``=``, while dlitz
          hashes contain no ``=`` at all.