PEP 476 -- Enabling certificate verification by default for stdlib http clients

 

 

SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain 

 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)

 

 https://docs.python.org/3.7/whatsnew/3.7.html#ssl

ssl¶

The ssl module now uses OpenSSL’s builtin API instead of match_hostname() to check a host name or an IP address. Values are validated during TLS handshake. Any certificate validation error including failing the host name check now raises SSLCertVerificationError and aborts the handshake with a proper TLS Alert message. The new exception contains additional information. Host name validation can be customized with SSLContext.hostname_checks_common_name. (Contributed by Christian Heimes in bpo-31399.)

Note

 

The improved host name check requires a libssl implementation compatible with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no longer supported (see Platform Support Removals for more details). The ssl module is mostly compatible with LibreSSL 2.7.2 and newer.

The ssl module no longer sends IP addresses in SNI TLS extension. (Contributed by Christian Heimes in bpo-32185.)

match_hostname() no longer supports partial wildcards like www*.example.org. (Contributed by Mandeep Singh in bpo-23033 and Christian Heimes in bpo-31399.)

The default cipher suite selection of the ssl module now uses a blacklist approach rather than a hard-coded whitelist. Python no longer re-enables ciphers that have been blocked by OpenSSL security updates. Default cipher suite selection can be configured at compile time. (Contributed by Christian Heimes in bpo-31429.)

Validation of server certificates containing internationalized domain names (IDNs) is now supported. As part of this change, the SSLSocket.server_hostname attribute now stores the expected hostname in A-label form ("xn--pythn-mua.org"), rather than the U-label form ("pythön.org"). (Contributed by Nathaniel J. Smith and Christian Heimes in bpo-28414.)

The ssl module has preliminary and experimental support for TLS 1.3 and OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still under development and TLS 1.3 hasn’t been finalized yet. The TLS 1.3 handshake and protocol behaves slightly differently than TLS 1.2 and earlier, see TLS 1.3. (Contributed by Christian Heimes in bpo-32947, bpo-20995, bpo-29136, bpo-30622 and bpo-33618)

SSLSocket and SSLObject no longer have a public constructor. Direct instantiation was never a documented and supported feature. Instances must be created with SSLContext methods wrap_socket() and wrap_bio(). (Contributed by Christian Heimes in bpo-32951)

OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are available as SSLContext.minimum_version and SSLContext.maximum_version. Supported protocols are indicated by several new flags, such as HAS_TLSv1_1. (Contributed by Christian Heimes in bpo-32609.)

Added SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake() to initiate TLS 1.3 post-handshake authentication. (Contributed by Christian Heimes in bpo-34670.)

 

 

PEP 476 -- Enabling certificate verification by default for stdlib http clients | Python.org https://www.python.org/dev/peps/pep-0476/

Python [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed 解决方法 - 微笑点燃希望 - 博客园 https://www.cnblogs.com/lykbk/p/ASDFQAWQWEQWEQWEQWEQWEQWEQEWEQW.html

PEP 476 -- Enabling certificate verification by default for stdlib http clients

PEP:	476
Title:	Enabling certificate verification by default for stdlib http clients
Author:	Alex Gaynor <alex.gaynor at gmail.com>
Status:	Final
Type:	Standards Track
Created:	28-August-2014
Resolution:	https://mail.python.org/pipermail/python-dev/2014-October/136676.html

---

Contents

• Abstract
• Rationale
• Technical Details
  • Trust database
  • Backwards compatibility
  • Opting out
• Other protocols
• Python Versions
• Implementation
• Copyright

Abstract

Currently when a standard library http client (the urllib, urllib2, http, and httplib modules) encounters an https:// URL it will wrap the network HTTP traffic in a TLS stream, as is necessary to communicate with such a server. However, during the TLS handshake it will not actually check that the server has an X509 certificate is signed by a CA in any trust root, nor will it verify that the Common Name (or Subject Alternate Name) on the presented certificate matches the requested host.

The failure to do these checks means that anyone with a privileged network position is able to trivially execute a man in the middle attack against a Python application using either of these HTTP clients, and change traffic at will.

This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.

Rationale

The "S" in "HTTPS" stands for secure. When Python's users type "HTTPS" they are expecting a secure connection, and Python should adhere to a reasonable standard of care in delivering this. Currently we are failing at this, and in doing so, APIs which appear simple are misleading users.

When asked, many Python users state that they were not aware that Python failed to perform these validations, and are shocked.

The popularity of requests (which enables these checks by default) demonstrates that these checks are not overly burdensome in any way, and the fact that it is widely recommended as a major security improvement over the standard library clients demonstrates that many expect a higher standard for "security by default" from their tools.

The failure of various applications to note Python's negligence in this matter is a source of regular CVE assignment [1] [2] [3] [4][5] [6] [7] [8] [9] [10] [11].

[1]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340

[2]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533

[3]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822

[4]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825

[5]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909

[6]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037

[7]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073

[8]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191

[9]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111

[10]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396

[11]	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444

Technical Details

Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it.

This will be achieved by adding a new ssl._create_default_https_context function, which is the same as ssl.create_default_context.

http.client can then replace its usage of ssl._create_stdlib_context with the ssl._create_default_https_context.

Additionally ssl._create_stdlib_context is renamed ssl._create_unverified_context (an alias is kept around for backwards compatibility reasons).

Trust database

This PEP proposes using the system-provided certificate database. Previous discussions have suggested bundling Mozilla's certificate database and using that by default. This was decided against for several reasons:

• Using the platform trust database imposes a lower maintenance burden on the Python developers -- shipping our own trust database would require doing a release every time a certificate was revoked.
• Linux vendors, and other downstreams, would unbundle the Mozilla certificates, resulting in a more fragmented set of behaviors.
• Using the platform stores makes it easier to handle situations such as corporate internal CAs.

OpenSSL also has a pair of environment variables, SSL_CERT_DIR and SSL_CERT_FILE which can be used to point Python at a different certificate database.

Backwards compatibility

This change will have the appearance of causing some HTTPS connections to "break", because they will now raise an Exception during handshake.

This is misleading however, in fact these connections are presently failing silently, an HTTPS URL indicates an expectation of confidentiality and authentication. The fact that Python does not actually verify that the user's request has been made is a bug, further: "Errors should never pass silently."

Nevertheless, users who have a need to access servers with self-signed or incorrect certificates would be able to do so by providing a context with custom trust roots or which disables validation (documentation should strongly recommend the former where possible). Users will also be able to add necessary certificates to system trust stores in order to trust them globally.

Twisted's 14.0 release made this same change, and it has been met with almost no opposition.

Opting out

For users who wish to opt out of certificate verification on a single connection, they can achieve this by providing the contextargument to urllib.urlopen:

import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

It is also possible, though highly discouraged, to globally disable verification by monkeypatching the ssl module in versions of Python that implement this PEP:

import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context

This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).

Particularly security sensitive applications should always provide an explicit application defined SSL context rather than relying on the default behaviour of the underlying Python implementation.

Other protocols

This PEP only proposes requiring this level of validation for HTTP clients, not for other protocols such as SMTP.

This is because while a high percentage of HTTPS servers have correct certificates, as a result of the validation performed by browsers, for other protocols self-signed or otherwise incorrect certificates are far more common. Note that for SMTP at least, this appears to be changing and should be reviewed for a potential similar PEP in the future:

• https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223
• https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526

Python Versions

This PEP describes changes that will occur on both the 3.4.x, 3.5 and 2.7.X branches. For 2.7.X this will require backporting the context (SSLContext) argument to httplib, in addition to the features already backported in PEP 466.

Implementation

• LANDED: Issue 22366 adds the context argument to urlib.request.urlopen.
• Issue 22417 implements the substance of this PEP.

Copyright

This document has been placed into the public domain.

Source: https://github.com/python/peps/blob/master/pep-0476.txt

 

https://www.openssl.org/source/openssl-1.1.1d.tar.gz

 

Following modules built successfully but were removed because they could not be imported:
_hashlib _ssl

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381