2024-03-29T03:56:36.422 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T03:56:36.422 DEBUG:teuthology.run:Teuthology command: teuthology --description powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-distros/centos_latest tasks/readwrite thrashosds-health} --archive /home/teuthworker/archive/teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi/7628890 --name teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose -- /home/teuthworker/archive/teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi/7628890/orig.config.yaml 2024-03-29T03:56:36.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T03:56:36.538 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi/7628890 branch: squid description: powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-distros/centos_latest tasks/readwrite thrashosds-health} email: ceph-qa@ceph.com first_in_suite: false job_id: '7628890' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_TRIM\) - \(MDS_SLOW_REQUEST\) - MDS_SLOW_METADATA_IO - Behind on trimming - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - mds.a - client.0 - - osd.0 - - osd.1 - - osd.2 seed: 6088 sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba sleep_before_teardown: 0 subset: 2/4 suite: powercycle suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZEzUaPO6wk7oZ1TFA+ugiuBrfnoH0JnaSZXeUIPE1MO0Hym+6XkM68t6vlyVE4ughsiaFekDbZCWB3/gc3H/Y= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDKQtLCo/wKWS5Gcdkx60v7tX1iEKoXBq1nZ5SiXmWCDyP6c0FVqvh++bUHpiXOgF0V8E//GNeXuDL7VQbcHWY= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHN0sJrnO6WsXwOqYolL4Zh2Ik5RzZZrasaznFvga8P1QqcJx6ltiCTfmAmSa34S79zdE6CmdbX8jj915lplksg= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWPhAtYviuzhJO7VkwnqL3RDKbYNrsCin0sYC7fKMpz9KQzEDIwmoNQsE1m1jOSCNOscybOnMO0aANEZJoPBJs= tasks: - install: extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - thrashosds: chance_down: 1.0 powercycle: true timeout: 600 - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_21:32:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T03:56:36.539 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa; will attempt to use it 2024-03-29T03:56:36.539 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks 2024-03-29T03:56:36.539 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T03:56:36.541 INFO:teuthology.task.internal:Checking packages... 2024-03-29T03:56:36.567 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8da23624bc17fe0b1f797227e72a0db0ce7821ba' 2024-03-29T03:56:36.567 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T03:56:36.568 INFO:teuthology.packaging:ref: None 2024-03-29T03:56:36.568 INFO:teuthology.packaging:tag: None 2024-03-29T03:56:36.568 INFO:teuthology.packaging:branch: squid 2024-03-29T03:56:36.568 INFO:teuthology.packaging:sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T03:56:36.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-29T03:56:36.828 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1959.g8da23624 2024-03-29T03:56:36.830 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T03:56:36.836 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T03:56:36.836 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T03:56:36.852 INFO:teuthology.task.internal:Saving configuration 2024-03-29T03:56:36.866 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T03:56:36.875 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T03:56:36.900 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629076', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:51:28.156004', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZEzUaPO6wk7oZ1TFA+ugiuBrfnoH0JnaSZXeUIPE1MO0Hym+6XkM68t6vlyVE4ughsiaFekDbZCWB3/gc3H/Y='} 2024-03-29T03:56:36.900 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/check_lock.py", line 30, in check_lock assert status['locked_by'] == ctx.owner, \ AssertionError: machine smithi097.front.sepia.ceph.com is locked by scheduled_mchangir@teuthology, not scheduled_teuthology@teuthology 2024-03-29T03:56:37.120 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4933bd38a70e439da69b2198224d44b7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/check_lock.py", line 30, in check_lock assert status['locked_by'] == ctx.owner, \ AssertionError: machine smithi097.front.sepia.ceph.com is locked by scheduled_mchangir@teuthology, not scheduled_teuthology@teuthology 2024-03-29T03:56:37.128 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T03:56:37.162 INFO:teuthology.nuke:Not nuking smithi097.front.sepia.ceph.com because description doesn't match: teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi != /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629076 2024-03-29T03:56:37.185 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-29T03:56:37.186 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T03:56:37.209 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi/7628890', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 03:46:27.323380', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDKQtLCo/wKWS5Gcdkx60v7tX1iEKoXBq1nZ5SiXmWCDyP6c0FVqvh++bUHpiXOgF0V8E//GNeXuDL7VQbcHWY='} 2024-03-29T03:56:37.212 INFO:teuthology.nuke:Not nuking smithi135.front.sepia.ceph.com because description doesn't match: teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi != /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629077 2024-03-29T03:56:37.232 INFO:teuthology.nuke:Not nuking smithi154.front.sepia.ceph.com because description doesn't match: teuthology-2024-03-28_21:32:02-powercycle-squid-distro-default-smithi != /home/teuthworker/archive/mchangir-2024-03-29_02:01:30-fs:functional-wip-mchangir-testing1-main-20240328-072412-distro-default-smithi/7629076 2024-03-29T03:56:37.258 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-29T03:56:37.259 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T03:56:37.285 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-29T03:56:37.285 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T03:56:37.308 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-29T03:56:41.310 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T03:56:41.420 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-29T03:56:45.422 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T03:56:45.532 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-29T03:56:45.633 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-29T03:56:45.735 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZEzUaPO6wk7oZ1TFA+ugiuBrfnoH0JnaSZXeUIPE1MO0Hym+6XkM68t6vlyVE4ughsiaFekDbZCWB3/gc3H/Y= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHN0sJrnO6WsXwOqYolL4Zh2Ik5RzZZrasaznFvga8P1QqcJx6ltiCTfmAmSa34S79zdE6CmdbX8jj915lplksg= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWPhAtYviuzhJO7VkwnqL3RDKbYNrsCin0sYC7fKMpz9KQzEDIwmoNQsE1m1jOSCNOscybOnMO0aANEZJoPBJs= 2024-03-29T03:56:45.740 INFO:teuthology.run:Summary data: description: powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-distros/centos_latest tasks/readwrite thrashosds-health} failure_reason: machine smithi097.front.sepia.ceph.com is locked by scheduled_mchangir@teuthology, not scheduled_teuthology@teuthology owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4933bd38a70e439da69b2198224d44b7 status: fail success: false 2024-03-29T03:56:45.741 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T03:56:45.829 INFO:teuthology.run:FAIL