2024-02-18T13:14:16.411 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-18T13:14:16.412 INFO:teuthology.results:seed: '9680' 2024-02-18T13:26:53.723 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-18T13:26:53.727 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-18T20:11:25.101 INFO:teuthology.results:Tests finished! gathering results... 2024-02-18T20:11:25.643 INFO:scrape:Found 6 jobs 2024-02-18T20:11:25.647 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565669/teuthology.log 2024-02-18T20:11:25.649 INFO:scrape:Found 4 distinct failure reasons 2024-02-18T20:11:25.649 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi203 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v17.2.5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2024-02-18T20:11:25.649 INFO:scrape:3 jobs: ['7565666', '7565665', '7565668'] 2024-02-18T20:11:25.649 INFO:scrape:suites intersection: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2024-02-18T20:11:25.649 INFO:scrape:suites union: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2024-02-18T20:11:25.649 INFO:scrape: 2024-02-18T20:11:25.650 INFO:scrape:Failure: "2024-02-18T14:00:00.000108+0000 mon.a (mon.0) 2246 : cluster [WRN] overall HEALTH_WARN Degraded data redundancy: 83/380 objects degraded (21.842%), 21 pgs degraded" in cluster log 2024-02-18T20:11:25.650 INFO:scrape:1 jobs: ['7565667'] 2024-02-18T20:11:25.650 INFO:scrape:suites: ['supported-all-distro/ubuntu_latest}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2024-02-18T20:11:25.650 INFO:scrape: 2024-02-18T20:11:25.650 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi106.front.sepia.ceph.com: ['type=AVC msg=audit(1708263858.785:9246): avc: denied { rmdir } for pid=47979 comm="userdel" name="cephadm" dev="sda1" ino=656189 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1'] 2024-02-18T20:11:25.650 INFO:scrape:1 jobs: ['7565664'] 2024-02-18T20:11:25.650 INFO:scrape:suites: ['supported-all-distro/centos_8}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2024-02-18T20:11:25.650 INFO:scrape: 2024-02-18T20:11:25.650 INFO:scrape:Dead: None 2024-02-18T20:11:25.650 INFO:scrape:1 jobs: ['7565669'] 2024-02-18T20:11:25.650 INFO:scrape: 2024-02-18T20:11:25.651 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/ failed: 5 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7565664] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} ----------------------------------------------------------------- time: 01:08:23 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565664/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565664/ SELinux denials found on ubuntu@smithi106.front.sepia.ceph.com: ['type=AVC msg=audit(1708263858.785:9246): avc: denied { rmdir } for pid=47979 comm="userdel" name="cephadm" dev="sda1" ino=656189 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir permissive=1'] [7565665] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 06:35:21 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565665/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565665/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d3f90b1c4ed433ea8b3fb45f8ce20fc Command failed (workunit test rbd/test_librbd_python.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v17.2.5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' [7565666] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 04:50:45 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565666/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565666/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3752c7d1ea474650b984c1d7d8352764 Command failed (workunit test rbd/test_librbd_python.sh) on smithi203 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v17.2.5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' [7565667] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 01:09:26 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565667/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565667/ "2024-02-18T14:00:00.000108+0000 mon.a (mon.0) 2246 : cluster [WRN] overall HEALTH_WARN Degraded data redundancy: 83/380 objects degraded (21.842%), 21 pgs degraded" in cluster log [7565668] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 04:40:17 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565668/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565668/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a166d7dd0a0e422686597361f9f48376 Command failed (workunit test rbd/test_librbd_python.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=v17.2.5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' Dead ================================================================= [7565669] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565669/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-02-18_01:35:06-upgrade:quincy-p2p-quincy-distro-default-smithi/7565669/ Error reimaging machines: Expected smithi079's OS to be ubuntu 20.04 but found ubuntu 22.04