pytest 插件测试实战:assert_outcomes 报错优化与终端摘要被插件破坏时的隔离方法

pytest 插件测试实战:assert_outcomes 报错优化与终端摘要被插件破坏时的隔离方法【免费下载链接】pytestThe pytest framework makes it easy to write small tests, yet scales to support complex functional testing项目地址: https://gitcode.com/GitHub_Trending/py/pytest本文基于 pytest 仓库的变更条目 13369.improvement.rst 展开,聚焦pytester插件测试工具链中一个高频痛点:RunResult.assert_outcomes在被测 pytest 子进程的终端摘要被插件修改或移除时抛出的错误信息如何变得更可诊断,以及如何在嵌套测试运行中隔离输出变更插件。读完后,你将理解assert_outcomes的完整解析链路、新错误信息的含义,并掌握-p no:plugin与PYTEST_DISABLE_PLUGIN_AUTOLOAD1两种隔离手段的正确用法。一、变更背景:pytester 为什么依赖终端摘要pytester是 pytest 内置的黑盒测试工具,专为测试 pytest 自身及其插件设计。它的核心工作流是:在隔离环境中生成临时测试文件与 conftest、执行一次嵌套的 pytest 运行、然后对输出做断言。这个工作流的入口是Pytester类,定义在 src/_pytest/pytester.py,其类文档明确写道:Facilities to write tests/configuration files, execute pytest in isolation, and match against expected output, perfect for black-box testing of pytest plugins.嵌套运行结束后会返回RunResult对象(定义于 src/_pytest/pytester.py),携带四个核心数据:ret:退出码,优先转换为ExitCode枚举;outlines/errlines:捕获的 stdout / stderr 行列表;duration:运行耗时;stdout/stderr:包装成LineMatcher的匹配器,供fnmatch_lines()等模式匹配方法使用。在 writing_plugins 指南中的典型用法是:def test_hello(pytester): Make sure that our plugin works. # create a temporary conftest.py file pytester.makeconftest( import pytest pytest.fixture(params[Brianna, Andreas, Floris]) def name(request): return request.param ) # create a temporary pytest test file pytester.makepyfile( def test_hello_default(hello): assert hello() Hello World! def test_hello_name(hello, name): assert hello(name) Hello {0}!.format(name) ) # run all tests with pytest result pytester.runpytest() # check that all 4 tests passed result.assert_outcomes(passed4)这里的关键一步是result.assert_outcomes(passed4)。它并不直接读取内部测试报告对象,而是解析嵌套运行打印到 stdout 的最后一行终端摘要,例如: 1 failed, 1 passed, 1 warning, 1 error in 0.13s assert_outcomes的完整签名为(见 src/_pytest/pytester.py):def assert_outcomes( self, passed: int 0, skipped: int 0, failed: int 0, errors: int 0, xpassed: int 0, xfailed: int 0, warnings: int | None None, deselected: int | None None, ) - None: Assert that the specified outcomes appear with the respective numbers (0 means it didnt occur) in the text output from a test run. warnings and deselected are only checked if not None. 各计数参数默认0(表示该结果未出现);warnings与deselected默认为None,表示不检查这两项。二、解析链路:parseoutcomes → parse_summary_nounsassert_outcomes的调用链是:assert_outcomes→parseoutcomes→parse_summary_nouns→ 独立的断言辅助函数。2.1 从倒数第一行开始寻找摘要parseoutcomes(见 src/_pytest/pytester.py)只是把self.outlines委托给类方法parse_summary_nouns,其核心逻辑如下:classmethod def parse_summary_nouns(cls, lines) - dict[str, int]: for line in reversed(lines): if rex_session_duration.search(line): outcomes rex_outcome.findall(line) ret {noun: int(count) for (count, noun) in outcomes} break else: raise ValueError( Pytest terminal summary report not found. Plugins that modify pytests terminal output can break outcome parsing. Disable the plugin for the test run, for example with -p no:plugin, or disable plugin autoloading with PYTEST_DISABLE_PLUGIN_AUTOLOAD1. ) to_plural { warning: warnings, error: errors, } return {to_plural.get(k, k): v for k, v in ret.items()}其中两条正则定义在模块顶部 src/_pytest/pytester.py:rex_session_duration re.compile(r\d\.\d\ds) # 匹配 0.13s 这类耗时 rex_outcome re.compile(r(\d) (\w)) # 提取 1 passed 这样的 计数名词也就是说,解析器从 stdout 的最后一行向前扫描,找到第一条包含\d\.\d\ds(带两位小数的秒数)的行,再用(\d) (\w)提取所有数字 名词对。for...else结构的含义是:遍历完所有行都没找到摘要行时,执行else分支抛出ValueError。解析结果会做单复数归一:warning → warnings、error → errors,其余名词(passed、failed、xfailed、xpassed、deselected等)保持原样,保证返回的字典键名始终为复数形式。2.2 断言逻辑放在独立插件里找到摘要后,实际比对发生在 src/_pytest/pytester_assertions.py:def assert_outcomes( outcomes: dict[str, int], passed: int 0, skipped: int 0, failed: int 0, errors: int 0, xpassed: int 0, xfailed: int 0, warnings: int | None None, deselected: int | None None, ) - None: Assert that the specified outcomes appear with the respective numbers (0 means it didnt occur) in the text output from a test run. __tracebackhide__ True ... assert obtained expected值得注意的是,这段代码不在 pytester 主模块内,而是单独放在pytester_assertions插件中,并由 src/_pytest/pytester.py 的pytest_plugins [pytester_assertions]加载。文件头注释解释了原因:This plugin contains assertions used by pytester. pytester cannot contain them itself, since it is imported by thepytestmodule, hence cannot be subject to assertion rewriting, which requires a module to not be already imported.因为pytester被pytest包直接导入,已导入的模块无法被断言重写(Assertion Rewriting)处理;把断言函数拆到独立插件里,才能让失败时的报错显示展开后的实际值而非不透明的字典比较。__tracebackhide__ True则让 traceback 跳过这一内部层,直接指向用户测试代码。三、本次改进:错误信息从找不到摘要到告诉你怎么办for...else分支中的ValueError正是本次变更(#13369)改进的对象。改进后的消息包含三段信息:现象:Pytest terminal summary report not found.—— stdout 中找不到符合\d\.\d\ds特征的终端摘要行;根因提示:Plugins that modify pytests terminal output can break outcome parsing.—— 明确指出最常见的诱因是修改了 pytest 终端输出的插件;可执行的处置方案:Disable the plugin for the test run, for example with-p no:, or disable plugin autoloading withPYTEST_DISABLE_PLUGIN_AUTOLOAD1.—— 直接给出两种隔离手段。这个错误在两类场景下会被触发,均可从 parseoutcomes 的文档字符串印证:插件通过pytest_terminal_summary等 hook 修改或完全覆盖了标准摘要行,导致耗时特征匹配失败;嵌套运行根本没有产生正常摘要(例如启动阶段就报错),此时 stdout 里同样不存在摘要行。方法文档也同步更新了约束说明(src/_pytest/pytester.py):This method requires pytests standard terminal summary; see :meth:parseoutcomes.这提醒使用者:assert_outcomes的契约是pytest 标准终端摘要必须存在,它是基于文本解析的弱耦合断言,而非读取内部报告对象。四、隔离输出变更插件的两种手段文档化部分给出的方案对应 pytest 插件发现机制中的两个拦截点。插件加载顺序在 writing_plugins 指南中有完整说明,这里摘录与隔离直接相关的两条:启动时扫描命令行的-p no:name选项并阻断对应插件加载(内置插件同样可被阻断),发生在正常命令行解析之前;通过安装包的 entry points 自动加载第三方插件,除非设置了PYTEST_DISABLE_PLUGIN_AUTOLOAD环境变量。4.1 手段一:-p no:plugin精准禁用单个插件当你知道是哪个插件改了摘要时,在嵌套运行的参数里直接传-p no:plugin即可:def test_my_plugin(pytester): pytester.makepyfile(def test_one(): pass) # 摘要变更来自 my_terminal_plugin,禁用它 result pytester.runpytest(-p, no:my_terminal_plugin) result.assert_outcomes(passed1)从源码看,-p no:的拦截发生在命令行解析的最早期(见 doc/en/how-to/writing_plugins.rst 中 Plugin discovery order at tool startup 一节),因此无论该插件是内置的还是第三方自动加载的,都会被挡住。4.2 手段二:PYTEST_DISABLE_PLUGIN_AUTOLOAD1全量关闭自动发现当嵌套运行根本不应该发现任何第三方插件(更彻底、也更贴近纯净环境)时,设置该环境变量:def test_isolated(pytester, monkeypatch): monkeypatch.setenv(PYTEST_DISABLE_PLUGIN_AUTOLOAD, 1) pytester.makepyfile(def test_one(): pass) result pytester.runpytest() result.assert_outcomes(passed1)环境变量的读取点在 src/_pytest/config/init.py 与 src/_pytest/config/init.py:bool(os.environ.get(PYTEST_DISABLE_PLUGIN_AUTOLOAD))为真时跳过 entry points 自动加载。它在 helpconfig 的--help环境变量列表中同样被列出(PYTEST_DISABLE_PLUGIN_AUTOLOAD Set to disable plugin auto-loading)。plugins 指南也给出了命令行形式的等价示例:PYTEST_DISABLE_PLUGIN_AUTOLOAD1 pytest -p xdist即关闭自动发现后,仍可用-p手动加载需要的插件,实现白名单式隔离。4.3 如何定位肇事插件排查思路可以按以下顺序推进:先看嵌套运行的完整输出:RunResult.stdout是LineMatcher,可直接print(result.stdout)检查最后一行是否还是标准摘要;对比禁用前后的行为:在runpytest参数中加-p no:plugin逐个排除,或用PYTEST_DISABLE_PLUGIN_AUTOLOAD1做一次基线运行;确认插件身份后,把隔离参数固化进测试,而不是每次手工复现。pytest 自己的测试套件也大量使用这套手法,例如 testing/test_terminal.py 中多处通过monkeypatch.delenv(PYTEST_DISABLE_PLUGIN_AUTOLOAD, raisingFalse)临时清掉外层环境的影响,再验证终端输出;testing/conftest.py 则全局设置PYTEST_DISABLE_PLUGIN_AUTOLOAD1保证外层测试环境的隔离性。五、回归验证:新错误信息有专门的测试用例本次改进在 testing/test_pytester.py 中有对应的回归测试:def test_assert_outcomes_after_pytest_error(pytester: Pytester) - None: pytester.makepyfile(def test_foo(): assert True) result pytester.runpytest(--unexpected-argument) with pytest.raises(ValueError) as exc_info: result.assert_outcomes(passed0) message str(exc_info.value) assert Plugins that modify pytests terminal output in message assert PYTEST_DISABLE_PLUGIN_AUTOLOAD1 in message测试路径是:嵌套运行传入一个非法参数--unexpected-argument,使 pytest 在启动阶段报错、不产生标准摘要;随后assert_outcomes必须抛出ValueError,且消息中必须包含Plugins that modify pytests terminal output和PYTEST_DISABLE_PLUGIN_AUTOLOAD1两个关键片段。这既锁定了错误的类型与触发条件,也锁定了新文案中的两条处置建议,防止后续重构时丢失可操作信息。六、实战清单:编写插件测试时如何避免踩坑结合上述源码与文档,插件作者的pytester测试可以遵循以下实践:优先使用assert_outcomes做结果断言,它比手写fnmatch_lines更简洁,且支持passed/skipped/failed/errors/xpassed/xfailed六类必检计数与warnings/deselected两类可选计数;在嵌套参数里显式隔离会改输出的插件:runpytest(-p, no:plugin)是成本最低的精准手段;环境级隔离用PYTEST_DISABLE_PLUGIN_AUTOLOAD1:适合 CI 中第三方插件不可控的场景,可配合-p显式加载白名单插件;失败时先读错误信息:ValueError文案已直接给出排查方向,若嵌套运行本就不应产生摘要(如启动报错),应改断言result.ret或result.stderr,而不是强行assert_outcomes;理解解析边界:parse_summary_nouns只匹配带\d\.\d\ds耗时特征的摘要行,自定义摘要若改变时间格式,同样会使assert_outcomes不可用——此时可用parse_summary_nouns之外的LineMatcher方法直接匹配输出文本。小结assert_outcomes是pytester黑盒测试工作流中对测试到底跑成了什么样最直接的断言手段,但它的前提是 pytest 标准终端摘要完好。#13369 这次改进把找不到摘要的失败从一句模糊的报错,升级为明确指出插件是常见诱因、并给出-p no:plugin与PYTEST_DISABLE_PLUGIN_AUTOLOAD1两条可操作路径的ValueError,同时在 writing_plugins 指南中补充了对应说明(src/_pytest/pytester.py 为改进后的实现,testing/test_pytester.py 为其回归测试)。对插件作者而言,这意味着:写pytester测试时,把隔离输出变更插件当作与断言本身同等重要的一部分来设计,嵌套运行的结果才可稳定复现。【免费下载链接】pytestThe pytest framework makes it easy to write small tests, yet scales to support complex functional testing项目地址: https://gitcode.com/GitHub_Trending/py/pytest创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考