openstack/nova
Code Issues Proposed changes
Files
88560094450bb4935909793965866b542d9ef3fe
nova/tox.ini
Chris Dent e5269f2fbb Correct lower-constraints.txt and the related tox job 
In the review of a similar change in placement [1], it was realized that
the nova lower-constraints tox job probably had the same problems.
Testing revealed this to be the case. This change fixes the job and
updates the related requirements problems accordingly.

The are two main factors at play here:

* The default install_command in tox.ini uses the upper_contraints.txt
  file. When there is more than one constraints.txt they are merged and
  the higher constraints win. Using upper and lower at the same time
  violates the point of lower (which is to indicate the bare minimum
  we are capable of using).

* When usedevelop is true in tox, the command that is run to install the
  current projects code is something like 'python setup.py develop',
  which installs a project's requirements _after_ the install_command has
  run, clobbering the constrained installs. When using pbr,
  'python setup.py install' (used when usedevelop is False) does not do
  this.

Fixing those then makes it possible to use the test to fix the
lower-constraints.txt and *requirements.txt files, changes include:

* Defining 'usedevelop = False' in the 'lower-constraints' target and
  removing the otherwise superfluous 'skipsdist' global setting to
  ensure requirements aren't clobbered.

* Removing packages which show up in lower-constraints.txt but not in
  the created virtualenv. Note that the job only runs unit tests, so
  this may be incomplete. In the placement version of this both unit and
  functional are run. We may want to consider that here.

* Updating cryptography. This version is needed with more recent
  pyopenssl.

* Updated keystonemiddleware. This is needed for some tests which
  confirm passing configuration to the middleware.

* Update psycopg2 to a version that can talk with postgresql 10.

* Add PyJWT, used by zVMCloudConnector

* Update zVMCloudConnector to a version that works with Python 3.5 and
  beyond.

* Update olso.messaging to versions that work with the tests, under
  Python 3.

* Adding missing transitive packages.

* Adjusting alpha-ordering to map to how pip freeze does it.

* setuptools is removed from requirements.txt because the created
  virtualenv doesn't contain it

NOTE: The lower-constraints.txt file makes no commitment to expressing
minimum requirements for anything other than the current basepython.
So the fact that a different set of lower-constraints would be present
if we were using python2 is not relevant. See discussion at [1].
However, since requirements.txt _is_ used for python2, the
requirements-check gate job requires that enum34 be present in
lower-constraints.txt because it is in requirements.txt.

NOTE: A test is removed because it cannot work in the
lower-constraints context: 'test_policy_generator_from_command_line'
forks a call to 'oslopolicy-policy-generator --namespace nova' which
fails because stevedore fails to pick up nova-based entry points when
in a different process. This is because of the change to usedevelop.
After discussion with the original author of the test removal was
considered an acceptable choice.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2019-03-05.log.html#t2019-03-05T13:28:23

Closes-Bug: #1822575

Change-Id: Ic6466b0440a4fe012731a63715cf5d793b6ae4dd
2019-04-01 19:16:33 +00:00

290 lines
9.8 KiB
INI
Raw Blame History

[tox]
minversion = 3.1.1
envlist = py{27,35},functional,pep8
# Automatic envs (pyXX) will use the python version appropriate to that
# env and ignore basepython inherited from [testenv]. That's what we
# want, and we don't need to be warned about it.
ignore_basepython_conflict = True
[testenv]
basepython = python3
usedevelop = True
whitelist_externals =
bash
find
rm
env
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
setenv =
VIRTUAL_ENV={envdir}
LANGUAGE=en_US
LC_ALL=en_US.utf-8
OS_STDOUT_CAPTURE=1
OS_STDERR_CAPTURE=1
OS_TEST_TIMEOUT=160
PYTHONDONTWRITEBYTECODE=1
# TODO(stephenfin): Remove psycopg2 when minimum constraints is bumped to 2.8
PYTHONWARNINGS = ignore::UserWarning:psycopg2
deps = -r{toxinidir}/test-requirements.txt
passenv =
OS_DEBUG GENERATE_HASHES
# there is also secret magic in subunit-trace which lets you run in a fail only
# mode. To do this define the TRACE_FAILONLY environmental variable.
[testenv:py27]
commands =
stestr run {posargs}
env TEST_OSPROFILER=1 stestr run --combine --no-discover 'nova.tests.unit.test_profiler'
stestr slowest
[testenv:py35]
commands =
stestr run {posargs}
env TEST_OSPROFILER=1 stestr run --combine --no-discover 'nova.tests.unit.test_profiler'
[testenv:py36]
commands =
{[testenv:py35]commands}
[testenv:py37]
commands =
{[testenv:py35]commands}
[testenv:pep8]
description =
Run style checks.
envdir = {toxworkdir}/shared
commands =
bash tools/flake8wrap.sh {posargs}
# Check that all JSON files don't have \r\n in line.
bash -c "! find doc/ -type f -name *.json | xargs grep -U -n $'\r'"
# Check that all included JSON files are valid JSON
bash -c '! find doc/ -type f -name *.json | xargs -t -n1 python -m json.tool 2>&1 > /dev/null | grep -B1 -v ^python'
[testenv:fast8]
description =
Run style checks on the changes made since HEAD~. For a full run including docs, use 'pep8'
envdir = {toxworkdir}/shared
commands =
bash tools/flake8wrap.sh -HEAD
[testenv:functional]
# TODO(melwitt): This can be removed when functional tests are gating with
# python 3.x
# NOTE(cdent): For a while, we shared functional virtualenvs with the unit
# tests, to save some time. However, this conflicts with tox siblings in zuul,
# and we need siblings to make testing against master of other projects work.
basepython = python2.7
setenv = {[testenv]setenv}
# As nova functional tests import the PlacementFixture from the placement
# repository these tests are, by default, set up to run with latest master from
# the placement repo. In the gate, Zuul will clone the latest master from
# placement OR the version of placement the Depends-On in the commit message
# suggests. If you want to run the test locally with an un-merged placement
# change, modify this line locally to point to your dependency or pip install
# placement into the appropriate tox virtualenv. We express the requirement
# here instead of test-requirements because we do not want placement present
# during unit tests.
deps =
-r{toxinidir}/test-requirements.txt
git+https://git.openstack.org/openstack/placement#egg=openstack-placement
commands =
# NOTE(cdent): The group_regex describes how stestr will group tests into the
# same process when running concurently. The following ensures that gabbi tests
# coming from the same YAML file are all in the same process. This is important
# because each YAML file represents an ordered sequence of HTTP requests. Note
# that tests which do not match this regex will not be grouped in any
# special way. See the following for more details.
# http://stestr.readthedocs.io/en/latest/MANUAL.html#grouping-tests
# https://gabbi.readthedocs.io/en/latest/#purpose
stestr --test-path=./nova/tests/functional --group_regex=nova\.tests\.functional\.api\.openstack\.placement\.test_placement_api(?:\.|_)([^_]+) run {posargs}
stestr slowest
# TODO(gcb) Merge this into [testenv:functional] when functional tests are gating
# with python 3.5
[testenv:functional-py35]
basepython = python3.5
setenv = {[testenv]setenv}
deps = {[testenv:functional]deps}
commands =
{[testenv:functional]commands}
[testenv:functional-py36]
basepython = python3.6
setenv = {[testenv]setenv}
deps = {[testenv:functional]deps}
commands =
{[testenv:functional]commands}
[testenv:functional-py37]
basepython = python3.7
setenv = {[testenv]setenv}
deps = {[testenv:functional]deps}
commands =
{[testenv:functional]commands}
[testenv:api-samples]
setenv =
{[testenv]setenv}
GENERATE_SAMPLES=True
PYTHONHASHSEED=0
deps = {[testenv:functional]deps}
commands =
stestr --test-path=./nova/tests/functional/api_sample_tests run {posargs}
stestr slowest
[testenv:genconfig]
envdir = {toxworkdir}/shared
commands =
oslo-config-generator --config-file=etc/nova/nova-config-generator.conf
[testenv:genpolicy]
envdir = {toxworkdir}/shared
commands =
oslopolicy-sample-generator --config-file=etc/nova/nova-policy-generator.conf
[testenv:genplacementpolicy]
envdir = {toxworkdir}/shared
commands =
oslopolicy-sample-generator --config-file=etc/nova/placement-policy-generator.conf
[testenv:cover]
# TODO(stephenfin): Remove the PYTHON hack below in favour of a [coverage]
# section once we rely on coverage 4.3+
#
# https://bitbucket.org/ned/coveragepy/issues/519/
envdir = {toxworkdir}/shared
setenv =
{[testenv]setenv}
PYTHON=coverage run --source nova --parallel-mode
commands =
coverage erase
stestr run {posargs}
coverage combine
coverage html -d cover
coverage xml -o cover/coverage.xml
coverage report
[testenv:debug]
envdir = {toxworkdir}/shared
commands =
oslo_debug_helper {posargs}
[testenv:venv]
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/doc/requirements.txt
commands =
{posargs}
[testenv:docs]
description =
Build main documentation.
deps = -r{toxinidir}/doc/requirements.txt
commands =
rm -rf doc/build
# Check that all JSON files don't have \r\n in line.
bash -c "! find doc/ -type f -name *.json | xargs grep -U -n $'\r'"
# Check that all included JSON files are valid JSON
bash -c '! find doc/ -type f -name *.json | xargs -t -n1 python -m json.tool 2>&1 > /dev/null | grep -B1 -v ^python'
sphinx-build -W -b html -d doc/build/doctrees doc/source doc/build/html
# Test the redirects. This must run after the main docs build
whereto doc/build/html/.htaccess doc/test/redirect-tests.txt
[testenv:api-guide]
description =
Generate the API guide. Called from CI scripts to test and publish to developer.openstack.org.
envdir = {toxworkdir}/docs
deps = {[testenv:docs]deps}
commands =
rm -rf api-guide/build
sphinx-build -W -b html -d api-guide/build/doctrees api-guide/source api-guide/build/html
[testenv:api-ref]
description =
Generate the API ref. Called from CI scripts to test and publish to developer.openstack.org.
envdir = {toxworkdir}/docs
deps = {[testenv:docs]deps}
commands =
rm -rf api-ref/build
sphinx-build -W -b html -d api-ref/build/doctrees api-ref/source api-ref/build/html
[testenv:releasenotes]
description =
Generate release notes.
envdir = {toxworkdir}/docs
deps = {[testenv:docs]deps}
commands =
rm -rf releasenotes/build
sphinx-build -W -b html -d releasenotes/build/doctrees releasenotes/source releasenotes/build/html
[testenv:all-docs]
description =
Build all documentation including API guides and refs.
envdir = {toxworkdir}/docs
deps = -r{toxinidir}/doc/requirements.txt
commands =
{[testenv:docs]commands}
{[testenv:api-guide]commands}
{[testenv:api-ref]commands}
{[testenv:releasenotes]commands}
[testenv:bandit]
# NOTE(browne): This is required for the integration test job of the bandit
# project. Please do not remove.
envdir = {toxworkdir}/shared
commands = bandit -r nova -x tests -n 5 -ll
[flake8]
# E125 is deliberately excluded. See
# https://github.com/jcrocholl/pep8/issues/126. It's just wrong.
#
# Most of the whitespace related rules (E12* and E131) are excluded
# because while they are often useful guidelines, strict adherence to
# them ends up causing some really odd code formatting and forced
# extra line breaks. Updating code to enforce these will be a hard sell.
#
# H405 is another one that is good as a guideline, but sometimes
# multiline doc strings just don't have a natural summary
# line. Rejecting code for this reason is wrong.
#
# E251 Skipped due to https://github.com/jcrocholl/pep8/issues/301
enable-extensions = H106,H203,H904
ignore = E121,E122,E123,E124,E125,E126,E127,E128,E129,E131,E251,H405
exclude = .venv,.git,.tox,dist,*lib/python*,*egg,build,tools/xenserver*,releasenotes
# To get a list of functions that are more complex than 25, set max-complexity
# to 25 and run 'tox -epep8'.
# 34 is currently the most complex thing we have
# TODO(jogo): get this number down to 25 or so
max-complexity=35
[hacking]
local-check-factory = nova.hacking.checks.factory
import_exceptions = nova.i18n
[testenv:bindep]
# Do not install any requirements. We want this to be fast and work even if
# system dependencies are missing, since it's used to tell you what system
# dependencies are missing! This also means that bindep must be installed
# separately, outside of the requirements files, and develop mode disabled
# explicitly to avoid unnecessarily installing the checked-out repo too (this
# further relies on "tox.skipsdist = True" above).
usedevelop = False
deps = bindep
commands =
bindep test
[testenv:lower-constraints]
# We need our own install command to avoid upper constraints being considered
# when making the environment. Set usedevelop to false to avoid pbr installing
# requirements for us so all requirements are installed in one call to pip.
usedevelop = False
install_command = pip install {opts} {packages}
deps =
-c{toxinidir}/lower-constraints.txt
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/requirements.txt
commands =
stestr run {posargs}
View Git Blame Copy Permalink