BrokenPipeError错误和python subprocess.run()超时参数在Windows上无效

news/2024/12/5 11:59:26/

1、问题的发现

  今天,一个在windows上运行良好的python脚本放到linux下报错,提示错误 BrokenPipeError: [Errno 32]Broken pipe。经调查是subprocess.run方法的timeout参数在linux上的表现和windows上不一致导致的。

try:ret = subprocess.run(cmd, shell=True, check=True, timeout=5,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except Exception as e:logging.debug(f"Runner FAIL")

2、问题描述

 为了描述这个问题,做了下面这个例子。subprocess.run调用了1个需要10s才能执行完的程序,但是却设定了1s的超时时间。理论上这段代码应该在1s后因超时退出,但事实并不如此。

import subprocess
import timet = time.perf_counter()
args = 'python -c "import time; time.sleep(10)"'
try: p = subprocess.run(args, shell=True, check=True,timeout=1,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except Exception as e: print(f"except is {e}")
print(f'coast:{time.perf_counter() - t:.8f}s')

 在windows上测试:

PS C:\Users\peng\Desktop> Get-ComputerInfo | select WindowsProductName, WindowsVersion, OsHardwareAbstractionLayerWindowsProductName WindowsVersion OsHardwareAbstractionLayer
------------------ -------------- --------------------------
Windows 10 Pro     2009           10.0.19041.2251
PS C:\Users\peng\Desktop> python  .\test_subprocess.py
except is Command 'python -c "import time; time.sleep(10)"' timed out after 1 seconds
coast:10.03642740s
PS C:\Users\peng\Desktop>

 在linux上测试:

21:51:31 wp@PowerEdge:~/bak$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
21:56:45 wp@PowerEdge:~/bak$ python test_subprocess.py
except is Command 'python -c "import time; time.sleep(10)"' timed out after 1 seconds
coast:1.00303393s
21:57:02 wp@PowerEdge:~/bak$

 可见,subprocess.run的timeout参数在windows下并没有生效。subprocess.run执行指定的命令,等待命令执行完成后返回一个包含执行结果的CompletedProcess类的实例。这个函数的原型为:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)
  • args:表示要执行的命令。必须是一个字符串,字符串参数列表。
  • stdin、stdout 和 stderr:子进程的标准输入、输出和错误。其值可以是 subprocess.PIPE、subprocess.DEVNULL、一个已经存在的文件描述符、已经打开的文件对象或者 None。subprocess.PIPE 表示为子进程创建新的管道。subprocess.DEVNULL 表示使用 os.devnull。默认使用的是 None,表示什么都不做。另外,stderr 可以合并到 stdout 里一起输出。
  • timeout:设置命令超时时间。如果命令执行时间超时,子进程将被杀死,并弹出 TimeoutExpired 异常。
  • check:如果该参数设置为 True,并且进程退出状态码不是 0,则弹 出 CalledProcessError 异常。
  • encoding: 如果指定了该参数,则 stdin、stdout 和 stderr 可以接收字符串数据,并以该编码方式编码。否则只接收 bytes 类型的数据。
  • shell:如果该参数为 True,将通过操作系统的 shell 执行指定的命令。

3、问题分析

  subprocess.run 会等待进程终止并处理TimeoutExpired异常。在POSIX上,异常对象包含读取部分的stdoutstderr字节。上面测试在windows上失效的主要问题是使用了shell模式,启动了管道,管道句柄可能由一个或多个后代进程继承(如通过shell=True),所以当超时发生时,即使关闭了shell程序,而由shell启动的其他程序,本例中是python程序依然在运行中,所以阻止了subprocess.run退出直至使用管道的所有进程退出。如果改为shell=False,则在windows上也出现1s的结果:

python  .\test_subprocess.py
except is Command 'python -c "import time; time.sleep(10)"' timed out after 1 seconds
coast:1.00460970s

  可以说这是windows实现上的一个缺陷,具体的可见:
[subprocess] run() sometimes ignores timeout in Windows · Issue #87512 · python/cpython · GitHub
[subprocess] run() sometimes ignores timeout in Windows #87512


subprocess.run() handles TimeoutExpired by terminating the process and waiting on it. On POSIX, the exception object contains the partially read stdout and stderr bytes. For example:

cmd = 'echo spam; echo eggs >&2; sleep 2'
try: p = subprocess.run(cmd, shell=True, capture_output=True,text=True, timeout=1)
except subprocess.TimeoutExpired as e: ex = e>>> ex.stdout, ex.stderr
(b'spam\n', b'eggs\n')

 On Windows, subprocess.run() has to finish reading output with a second communicate() call, after which it manually sets the exception's stdout and stderr attributes.
The poses the problem that the second communicate() call may block indefinitely, even though the child process has terminated.
 The primary issue is that the pipe handles may be inherited by one or more descendant processes (e.g. via shell=True), which are all regarded as potential writers that keep the pipe from closing. Reading from an open pipe that's empty will block until data becomes available. This is generally desirable for efficiency, compared to polling in a loop. But in this case, the downside is that run() in Windows will effectively ignore the given timeout.
 Another problem is that _communicate() writes the input to stdin on the calling thread with a single write() call. If the input exceeds the pipe capacity (4 KiB by default -- but a pipesize 'suggested' size could be supported), the write will block until the child process reads the excess data. This could block indefinitely, which will effectively ignore a given timeout. The POSIX implementation, in contrast, correctly handles a timeout in this case.
 Also, Popen.exit() closes the stdout, stderr, and stdin files without regard to the _communicate() worker threads. This may seem innocuous, but if a worker thread is blocked on synchronous I/O with one of these files, WinAPI CloseHandle() will also block if it's closing the last handle for the file in the current process. (In this case, the kernel I/O manager has a close procedure that waits to acquire the file for the current thread before performing various housekeeping operations, primarily in the filesystem, such as clearing byte-range locks set by the current process.) A blocked close() is easy to demonstrate. For example:

args = 'python -c "import time; time.sleep(99)"'
p = subprocess.Popen(args, shell=True, stdout=subprocess.PIPE)
try: p.communicate(timeout=1)
except: passp.kill() # terminates the shell process -- not python.exe
with p: pass # stdout.close() blocks until python.exe exits

The Windows implementation of Popen._communicate() could be redesigned as follows:

  • read in chunks, with a size from 1 byte up to the maximum available,
    as determined by _winapi.PeekNamedPipe()
  • write to the child's stdin on a separate thread
  • after communicate() has started, ensure that synchronous I/O in worker
    threads has been canceled via CancelSynchronousIo() before closing
    the pipes.
    The _winapi module would need to wrap OpenThread() and CancelSynchronousIo(), plus define the TERMINATE_THREAD (0x0001) access right.

With the proposed changes, subprocess.run() would no longer special case TimeoutExpired on Windows.


http://www.ppmy.cn/news/540.html

相关文章

SpringBoot 3.0 新特性,内置声明式HTTP客户端

http interface 从 Spring 6 和 Spring Boot 3 开始,Spring 框架支持将远程 HTTP 服务代理成带有特定注解的 Java http interface。类似的库,如 OpenFeign 和 Retrofit 仍然可以使用,但 http interface 为 Spring 框架添加内置支持。 什么是…

基于rsync daemon 实现 sersync——sersync实现实时数据同步

1 sersync 介绍 sersync类似于inotify,同样用于监控,但它克服了inotify的缺点. inotify最大的不足是会产生重复事件,或者同一个目录下多个文件的操作会产生多个事件,例如,当监控目录中有5个文件时,删除目录…

pycharm please specify a different SDK name

虚拟环境重名,建议将所有虚拟环境删除,重新配置

Discourse 为什不建议使用 Gmail 的 SMTP

最开始我们也用了 Gmail 的 SMTP 服务。 这里有个问题是 Gmail 的日发送邮件限制,很多人可能认为 Gmail 是没有日常发送邮件限制的,通常不是这样的,因为如果你是手工回复和发送邮件的话,这个限制还是很难达到的。 如果是计算机或…

【Spring(七)】带你手写一个Spring容器

有关Spring的所有文章都收录于我的专栏:👉Spring👈 目录 前置准备 第一步、创建我们自定的注解 第二步、创建我们自己的容器类 测试 总结 相关文章 【Spring(一)】如何获取对象(Bean)【Spring&a…

构建基于 Ingress 的全链路灰度能力

作者:涂鸦 背景 随着云原生技术不断普及,越来越多的业务应用开始向云原生架构转变,借助容器管理平台 Kubernetes 的不可变基础设施、弹性扩缩容和高扩展性,助力业务迅速完成数字化转型。其中,集群入口流量管理方式在…

[附源码]Python计算机毕业设计Django高校社团管理系统

项目运行 环境配置: Pychram社区版 python3.7.7 Mysql5.7 HBuilderXlist pipNavicat11Djangonodejs。 项目技术: django python Vue 等等组成,B/S模式 pychram管理等等。 环境需要 1.运行环境:最好是python3.7.7,…

【数据库实验(6)】完整性约束

create table student(sno varchar(20),sname varchar(20),sgender varchar(10),sage int,sdept varchar(20))insert into student values(1222,张丽,女,21,计算机),(1223,刘立,男,25,数学),(1224,王晓华,女,22,计算机),(1225,李佳,男,23,计算机),(1226,张月,女,22,建筑)select…