2023-07-12T23:26:25.090 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-12T23:26:25.091 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dis@teuthology --archive /home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335449 --description krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} --name dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi --verbose -- /home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335449/orig.config.yaml 2023-07-12T23:26:25.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-12T23:26:25.204 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335449 branch: main description: krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} email: idryomov@gmail.com first_in_suite: false job_id: '7335449' kernel: flavor: default kdb: true sha1: 6b44f5b67a24f719e26c0df425dff33b2a18153e last_in_suite: false machine_type: smithi name: dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: main ceph: conf: client: rbd default features: 37 rbd default map options: ms_mode=legacy,rxbounce global: mon client directed command retry: 5 ms die on skipped message: false ms inject socket failures: 500 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 05e449f9a2a65c297f31628af8f01f63cf36f261 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 05e449f9a2a65c297f31628af8f01f63cf36f261 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: ed7555b1c51be1bc07c48524614abdf56e3a4072 owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph roles: - - mon.a - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.y - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 seed: 3308 sha1: 05e449f9a2a65c297f31628af8f01f63cf36f261 sleep_before_teardown: 0 subset: 1/99 suite: krbd suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_ed7555b1c51be1bc07c48524614abdf56e3a4072/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: ed7555b1c51be1bc07c48524614abdf56e3a4072 targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= smithi177.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - ceph: null - rbd: all: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-12_22:16:14 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-12T23:26:25.205 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_ed7555b1c51be1bc07c48524614abdf56e3a4072/qa; will attempt to use it 2023-07-12T23:26:25.205 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_ed7555b1c51be1bc07c48524614abdf56e3a4072/qa/tasks 2023-07-12T23:26:25.206 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-12T23:26:25.207 INFO:teuthology.task.internal:Checking packages... 2023-07-12T23:26:25.237 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '05e449f9a2a65c297f31628af8f01f63cf36f261' 2023-07-12T23:26:25.238 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-12T23:26:25.238 INFO:teuthology.packaging:ref: None 2023-07-12T23:26:25.238 INFO:teuthology.packaging:tag: None 2023-07-12T23:26:25.239 INFO:teuthology.packaging:branch: main 2023-07-12T23:26:25.239 INFO:teuthology.packaging:sha1: 05e449f9a2a65c297f31628af8f01f63cf36f261 2023-07-12T23:26:25.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=main 2023-07-12T23:26:26.021 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4869-g05e449f9-1focal 2023-07-12T23:26:26.021 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-12T23:26:26.028 INFO:teuthology.task.internal:no buildpackages task found 2023-07-12T23:26:26.029 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-12T23:26:26.041 INFO:teuthology.task.internal:Saving configuration 2023-07-12T23:26:26.054 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-12T23:26:26.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T23:26:26.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:55:47.454090', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-12T23:26:26.209 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:55:47.455734', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-12T23:26:26.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:17:56.456735', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-07-12T23:26:26.233 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-12T23:26:26.270 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-07-12T23:26:26.271 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-07-12T23:26:26.271 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['client.0'] 2023-07-12T23:26:26.271 INFO:teuthology.run_tasks:Running task console_log... 2023-07-12T23:26:26.469 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f92cd691160>, signals=[15]) 2023-07-12T23:26:26.471 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-12T23:26:26.478 INFO:teuthology.task.internal:Opening connections... 2023-07-12T23:26:26.479 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2023-07-12T23:26:26.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T23:26:26.584 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2023-07-12T23:26:26.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T23:26:26.688 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2023-07-12T23:26:26.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-12T23:26:26.739 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-12T23:26:26.746 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2023-07-12T23:26:26.766 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2023-07-12T23:26:26.766 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2023-07-12T23:26:26.824 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu" 2023-07-12T23:26:26.824 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-12T23:26:26.824 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu 2023-07-12T23:26:26.824 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="20.04" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=focal 2023-07-12T23:26:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=focal 2023-07-12T23:26:26.827 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2023-07-12T23:26:26.849 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2023-07-12T23:26:26.863 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2023-07-12T23:26:26.864 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2023-07-12T23:26:26.927 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="Ubuntu" 2023-07-12T23:26:26.927 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-12T23:26:26.927 INFO:teuthology.orchestra.run.smithi125.stdout:ID=ubuntu 2023-07-12T23:26:26.927 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE=debian 2023-07-12T23:26:26.927 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="20.04" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_CODENAME=focal 2023-07-12T23:26:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:UBUNTU_CODENAME=focal 2023-07-12T23:26:26.929 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-07-12T23:26:26.949 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2023-07-12T23:26:26.955 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2023-07-12T23:26:26.955 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2023-07-12T23:26:27.002 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="Ubuntu" 2023-07-12T23:26:27.002 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-07-12T23:26:27.002 INFO:teuthology.orchestra.run.smithi177.stdout:ID=ubuntu 2023-07-12T23:26:27.002 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE=debian 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="20.04" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://www.ubuntu.com/" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-07-12T23:26:27.003 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_CODENAME=focal 2023-07-12T23:26:27.004 INFO:teuthology.orchestra.run.smithi177.stdout:UBUNTU_CODENAME=focal 2023-07-12T23:26:27.004 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2023-07-12T23:26:27.032 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-12T23:26:27.041 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-12T23:26:27.077 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-12T23:26:27.077 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2023-07-12T23:26:27.084 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2023-07-12T23:26:27.091 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2023-07-12T23:26:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T23:26:27.093 ERROR:teuthology.task.internal:Host smithi043 has stale test directory /home/ubuntu/cephtest, check lock and cleanup. 2023-07-12T23:26:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-12T23:26:27.101 ERROR:teuthology.task.internal:Host smithi125 has stale test directory /home/ubuntu/cephtest, check lock and cleanup. 2023-07-12T23:26:27.101 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 299, in check_conflict raise RuntimeError('Stale jobs detected, aborting.') RuntimeError: Stale jobs detected, aborting. 2023-07-12T23:26:27.159 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bad28c42087b4bdf83811fa699757590 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 299, in check_conflict raise RuntimeError('Stale jobs detected, aborting.') RuntimeError: Stale jobs detected, aborting. 2023-07-12T23:26:27.161 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-12T23:26:27.327 INFO:teuthology.nuke:Checking targets against current locks 2023-07-12T23:26:27.445 DEBUG:teuthology.nuke:shortname: smithi043 2023-07-12T23:26:27.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T23:26:27.470 DEBUG:teuthology.nuke:shortname: smithi125 2023-07-12T23:26:27.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T23:26:27.474 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:55:47.454090', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-07-12T23:26:27.494 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:55:47.455734', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-12T23:26:27.500 DEBUG:teuthology.nuke:shortname: smithi177 2023-07-12T23:26:27.501 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-12T23:26:27.522 INFO:teuthology.orchestra.console:Power off smithi043 2023-07-12T23:26:27.523 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-12T23:26:27.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-12_22:16:14-krbd-main-wip-frame-segment-len-default-smithi/7335449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 22:17:56.456735', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-12T23:26:27.544 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:27.569 INFO:teuthology.orchestra.console:Power off smithi125 2023-07-12T23:26:27.570 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-12T23:26:27.593 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:27.609 INFO:teuthology.orchestra.console:Power off smithi177 2023-07-12T23:26:27.610 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-12T23:26:27.631 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:31.581 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:31.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:31.652 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:35.697 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:35.730 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:35.762 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:39.813 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:39.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:39.873 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-12T23:26:40.025 INFO:teuthology.orchestra.console:Power off for smithi043 completed 2023-07-12T23:26:40.051 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2023-07-12T23:26:40.082 INFO:teuthology.orchestra.console:Power off for smithi177 completed 2023-07-12T23:26:40.186 INFO:teuthology.run:Summary data: description: krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} failure_reason: Stale jobs detected, aborting. owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bad28c42087b4bdf83811fa699757590 status: fail success: false 2023-07-12T23:26:40.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-12T23:26:40.353 INFO:teuthology.run:FAIL