2024-02-10T05:12:24.221 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-10T05:12:24.221 INFO:teuthology.results:subset: '111/120000' 2024-02-10T05:12:24.222 INFO:teuthology.results:seed: '9514' 2024-02-10T05:19:09.569 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-10T05:19:09.573 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-10T09:02:26.791 INFO:teuthology.results:Tests finished! gathering results... 2024-02-10T09:02:27.393 INFO:scrape:Found 9 jobs 2024-02-10T09:02:31.484 INFO:scrape:Found 2 distinct failure reasons 2024-02-10T09:02:31.484 INFO:scrape:Failure: "2024-02-10T05:48:55.096013+0000 mon.a (mon.0) 74 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log 2024-02-10T09:02:31.484 INFO:scrape:7 jobs: ['7555066', '7555068', '7555065', '7555061', '7555063', '7555062', '7555067'] 2024-02-10T09:02:31.485 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2024-02-10T09:02:31.485 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-02-10T09:02:31.485 INFO:scrape: 2024-02-10T09:02:31.485 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2024-02-10T09:02:31.485 INFO:scrape:2 jobs: ['7555064', '7555060'] 2024-02-10T09:02:31.485 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-02-10T09:02:31.485 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default'] 2024-02-10T09:02:31.485 INFO:scrape: 2024-02-10T09:02:31.486 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7555060] upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:45:41 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555060/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78910ec7e18a448f9c82158ab184e0df Command failed (workunit test rbd/test_librbd_python.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7555061] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:55:15 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555061/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555061/ "2024-02-10T05:42:50.347895+0000 mon.a (mon.0) 99 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7555062] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:36:53 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555062/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555062/ "2024-02-10T05:37:28.732532+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7555063] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:33:05 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555063/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555063/ "2024-02-10T05:46:28.689357+0000 mon.a (mon.0) 107 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7555064] upgrade:reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:45:14 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555064/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555064/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cda31b57273440394c0ce0a689ee49c Command failed (workunit test rbd/test_librbd_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7555065] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:12:00 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555065/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555065/ "2024-02-10T05:50:00.000136+0000 mon.a (mon.0) 669 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7555066] upgrade:reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:05:56 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555066/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555066/ "2024-02-10T05:48:55.096013+0000 mon.a (mon.0) 74 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7555067] upgrade:reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:09:12 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555067/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555067/ "2024-02-10T05:41:19.325834+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7555068] upgrade:reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:45:34 info: https://pulpito.ceph.com/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555068/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-02-09_19:43:59-upgrade:reef-x-main-distro-default-smithi/7555068/ "2024-02-10T06:00:00.000129+0000 mon.a (mon.0) 828 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log