2022-04-01T19:01:49.098 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-01T19:01:49.228 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi079.front.sepia.ceph.com' reimaging 2022-04-01T19:01:49.230 INFO:teuthology.lock.ops:Updating [ubuntu@smithi079.front.sepia.ceph.com]: reset os type and version on server 2022-04-01T19:01:49.230 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2022-04-01T19:01:49.252 INFO:teuthology.lock.ops:Node 'ubuntu@smithi079.front.sepia.ceph.com' reimaging is complete 2022-04-01T19:01:49.252 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi188.front.sepia.ceph.com' reimaging 2022-04-01T19:01:49.254 INFO:teuthology.lock.ops:Updating [ubuntu@smithi188.front.sepia.ceph.com]: reset os type and version on server 2022-04-01T19:01:49.254 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2022-04-01T19:01:49.276 INFO:teuthology.lock.ops:Node 'ubuntu@smithi188.front.sepia.ceph.com' reimaging is complete 2022-04-01T19:01:49.468 INFO:teuthology.provision.fog.smithi079:Scheduling deploy of centos 8.stream 2022-04-01T19:01:49.481 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of centos 8.stream 2022-04-01T19:01:49.805 INFO:teuthology.orchestra.console:Power off smithi188 2022-04-01T19:01:49.833 INFO:teuthology.orchestra.console:Power off smithi079 2022-04-01T19:01:58.213 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2022-04-01T19:01:58.218 INFO:teuthology.orchestra.console:Power off for smithi079 completed 2022-04-01T19:01:58.322 INFO:teuthology.orchestra.console:Power on smithi188 2022-04-01T19:01:58.330 INFO:teuthology.orchestra.console:Power on smithi079 2022-04-01T19:02:06.694 INFO:teuthology.orchestra.console:Power on for smithi188 completed 2022-04-01T19:02:06.704 INFO:teuthology.orchestra.console:Power on for smithi079 completed 2022-04-01T19:02:06.796 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2022-04-01T19:02:06.805 INFO:teuthology.provision.fog.smithi079:Waiting for deploy to finish 2022-04-01T19:06:45.412 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-01T19:06:45.657 INFO:teuthology.provision.fog.smithi188:Node is ready 2022-04-01T19:06:45.715 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188.front.sepia.ceph.com 2022-04-01T19:06:45.773 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188 2022-04-01T19:06:46.119 INFO:teuthology.provision.fog.smithi188:Deploy complete! 2022-04-01T19:06:48.001 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-01T19:06:48.222 INFO:teuthology.provision.fog.smithi079:Node is ready 2022-04-01T19:06:48.280 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079.front.sepia.ceph.com 2022-04-01T19:06:48.339 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.15.79 smithi079.front.sepia.ceph.com smithi079 2022-04-01T19:06:48.683 INFO:teuthology.provision.fog.smithi079:Deploy complete! 2022-04-01T19:06:48.837 INFO:teuthology.lock.ops:Checking smithi079.front.sepia.ceph.com 2022-04-01T19:06:48.838 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-01T19:06:48.858 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com 2022-04-01T19:06:48.859 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-01T19:06:48.877 INFO:teuthology.lock.ops:Updating [smithi079.front.sepia.ceph.com]: set os type and version on server 2022-04-01T19:06:48.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2022-04-01T19:06:49.068 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2022-04-01T19:06:49.124 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2022-04-01T19:06:49.125 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="8" 2022-04-01T19:06:49.125 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2022-04-01T19:06:49.125 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2022-04-01T19:06:49.125 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="8" 2022-04-01T19:06:49.126 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el8" 2022-04-01T19:06:49.126 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-01T19:06:49.126 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2022-04-01T19:06:49.126 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-01T19:06:49.127 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2022-04-01T19:06:49.127 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-01T19:06:49.127 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-01T19:06:49.127 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-01T19:06:49.128 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2022-04-01T19:06:49.147 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server 2022-04-01T19:06:49.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2022-04-01T19:06:49.353 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2022-04-01T19:06:49.412 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2022-04-01T19:06:49.412 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2022-04-01T19:06:49.413 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2022-04-01T19:06:49.413 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2022-04-01T19:06:49.413 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2022-04-01T19:06:49.413 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2022-04-01T19:06:49.413 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-01T19:06:49.414 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2022-04-01T19:06:49.414 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-01T19:06:49.414 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2022-04-01T19:06:49.414 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-01T19:06:49.415 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-01T19:06:49.415 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-01T19:06:49.416 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2022-04-01T19:06:49.519 INFO:teuthology.dispatcher.supervisor:Running job 6773033 2022-04-01T19:06:49.519 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-04-01_18:03:43-fs-wip-yuri-testing-2022-04-01-0816-distro-default-smithi/6773033 --name yuriw-2022-04-01_18:03:43-fs-wip-yuri-testing-2022-04-01-0816-distro-default-smithi --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} -- /home/teuthworker/archive/yuriw-2022-04-01_18:03:43-fs-wip-yuri-testing-2022-04-01-0816-distro-default-smithi/6773033/orig.config.yaml 2022-04-01T19:06:49.522 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-04-01_18:03:43-fs-wip-yuri-testing-2022-04-01-0816-distro-default-smithi/6773033 2022-04-01T19:06:49.523 INFO:teuthology.dispatcher.supervisor:Job PID: 17020 2022-04-01T19:06:49.524 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-04-02T01:37:18.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-04-02T01:37:19.019 INFO:teuthology.kill:Killing Pids: {17020} 2022-04-02T01:37:20.182 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-04-02T01:37:20.183 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2022-04-02T01:37:20.184 INFO:teuthology.misc:Compressing logs... 2022-04-02T01:37:20.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2022-04-02T01:37:20.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2022-04-14T22:06:43.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 143, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 192, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 163, in gevent._gevent_cevent.Event.wait File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=12903) 2022-04-14T22:06:43.948 INFO:teuthology.kill:No teuthology processes running 2022-04-14T22:06:45.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2022-04-14T22:07:03.964 ERROR:teuthology.dispatcher.supervisor:run_with_watchdog had an unhandled exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 138, in run_job run_with_watchdog(p, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 301, in run_with_watchdog time.sleep(teuth_config.watchdog_interval) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/gevent/hub.py", line 166, in sleep hub.wait(t) File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=12903) 2022-04-14T22:07:03.967 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 61, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 138, in run_job run_with_watchdog(p, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 301, in run_with_watchdog time.sleep(teuth_config.watchdog_interval) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/gevent/hub.py", line 166, in sleep hub.wait(t) File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=12903)