2024-03-29T03:57:06.950 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-29T03:57:06.950 INFO:teuthology.results:subset: '19/32' 2024-03-29T03:57:06.950 INFO:teuthology.results:seed: '4408' 2024-03-29T04:17:37.018 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-29T04:17:37.021 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-29T08:02:55.379 INFO:teuthology.results:Tests finished! gathering results... 2024-03-29T08:02:58.512 INFO:scrape:Found 40 jobs 2024-03-29T08:02:59.950 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629124/teuthology.log 2024-03-29T08:03:22.988 INFO:scrape:Found 11 distinct failure reasons 2024-03-29T08:03:22.988 INFO:scrape:Failure: Command failed on smithi067 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe206bda-ed85-11ee-95d0-87774f69a715 -e sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba -- bash -c 'ceph orch ps'" 2024-03-29T08:03:22.988 INFO:scrape:2 jobs: ['7629126', '7629102'] 2024-03-29T08:03:22.988 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-03-29T08:03:22.989 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-03-29T08:03:22.989 INFO:scrape: 2024-03-29T08:03:22.989 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-03-29T08:03:22.989 INFO:scrape:6 jobs: ['7629110', '7629121', '7629117', '7629108', '7629098', '7629109'] 2024-03-29T08:03:22.989 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2024-03-29T08:03:22.989 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-03-29T08:03:22.989 INFO:scrape: 2024-03-29T08:03:22.989 INFO:scrape:Failure: "2024-03-29T05:10:00.000191+0000 mon.a (mon.0) 1714 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log 2024-03-29T08:03:22.989 INFO:scrape:10 jobs: ['7629127', '7629120', '7629113', '7629103', '7629111', '7629097', '7629123', '7629099', '7629092', '7629116'] 2024-03-29T08:03:22.990 INFO:scrape:suites intersection: [] 2024-03-29T08:03:22.990 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '1-tasks}', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{ubuntu_22.04}', 'workload/{ec-rados-default'] 2024-03-29T08:03:22.990 INFO:scrape: 2024-03-29T08:03:22.990 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific 2024-03-29T08:03:22.990 INFO:scrape:1 jobs: ['7629093'] 2024-03-29T08:03:22.990 INFO:scrape:suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream_runc}'] 2024-03-29T08:03:22.990 INFO:scrape: 2024-03-29T08:03:22.990 INFO:scrape:Dead: None 2024-03-29T08:03:22.990 INFO:scrape:3 jobs: ['7629124', '7629128', '7629100'] 2024-03-29T08:03:22.991 INFO:scrape: 2024-03-29T08:03:22.991 INFO:scrape:Failure: failed to install new kernel version within timeout 2024-03-29T08:03:22.991 INFO:scrape:1 jobs: ['7629101'] 2024-03-29T08:03:22.991 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream'] 2024-03-29T08:03:22.991 INFO:scrape: 2024-03-29T08:03:22.991 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-29T08:03:22.991 INFO:scrape:1 jobs: ['7629115'] 2024-03-29T08:03:22.991 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-03-29T08:03:22.991 INFO:scrape: 2024-03-29T08:03:22.991 INFO:scrape:Failure: "1711687038.104081 mon.a (mon.0) 366 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2024-03-29T08:03:22.991 INFO:scrape:2 jobs: ['7629114', '7629095'] 2024-03-29T08:03:22.992 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2024-03-29T08:03:22.992 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'workload/{ec-rados-default'] 2024-03-29T08:03:22.992 INFO:scrape: 2024-03-29T08:03:22.992 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-29T08:03:22.992 INFO:scrape:3 jobs: ['7629119', '7629107', '7629094'] 2024-03-29T08:03:22.992 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-03-29T08:03:22.992 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-03-29T08:03:22.992 INFO:scrape: 2024-03-29T08:03:22.992 INFO:scrape:Failure: Command failed on smithi027 with status 1: 'sudo yum install -y kernel' 2024-03-29T08:03:22.992 INFO:scrape:2 jobs: ['7629104', '7629106'] 2024-03-29T08:03:22.993 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2024-03-29T08:03:22.993 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc'] 2024-03-29T08:03:22.993 INFO:scrape: 2024-03-29T08:03:22.993 INFO:scrape:Failure: Command failed on smithi003 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.1d deep-scrub' 2024-03-29T08:03:22.993 INFO:scrape:2 jobs: ['7629125', '7629105'] 2024-03-29T08:03:22.993 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap'] 2024-03-29T08:03:22.993 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap'] 2024-03-29T08:03:22.993 INFO:scrape: 2024-03-29T08:03:22.996 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/ failed: 30 dead: 3 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7629092] upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:59:18 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629092/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629092/ "2024-03-29T04:38:37.922412+0000 mon.a (mon.0) 177 : cluster 4 [ERR] OSD_UNREACHABLE: 8 osds(s) are not reachable" in cluster log [7629093] upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:04:23 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629093/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f037dffaaa70481384f118d419012c0e Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci fic [7629094] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:32 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629094/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629094/ reached maximum tries (51) after waiting for 300 seconds [7629095] upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:27:58 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629095/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629095/ "1711685997.9097536 mon.a (mon.0) 667 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7629097] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:20:13 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629097/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629097/ "2024-03-29T04:39:55.808393+0000 mon.a (mon.0) 16 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7629098] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:52:16 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629098/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629098/ sentry: https://sentry.ceph.com/organizations/ceph/?query=72b3af3cd98646049f1abf18c8538e79 timeout expired in wait_until_healthy [7629099] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:07:11 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629099/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629099/ "2024-03-29T04:40:00.000147+0000 mon.a (mon.0) 575 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7629101] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629101/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c169345418d84eed8d12432e363165cc failed to install new kernel version within timeout [7629102] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:33 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629102/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629102/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21aa99c0345548a2977c745d057f2d47 Command failed on smithi055 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8045a158-ed85-11ee-95d0-87774f69a715 -e sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba -- bash -c 'ceph orch ps'" [7629103] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:40:16 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629103/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629103/ "2024-03-29T06:10:00.000116+0000 mon.a (mon.0) 7627 : cluster 4 [ERR] OSD_SCRUB_ERRORS: 5 scrub errors" in cluster log [7629104] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629104/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fde638a08654c1b97690801639baeb8 Command failed on smithi027 with status 1: 'sudo yum install -y kernel' [7629105] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:29:41 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629105/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629105/ Command failed on smithi033 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub' [7629106] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629106/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33f4e26264064429b524b2a2b79fc2b4 Command failed on smithi100 with status 1: 'sudo yum install -y kernel' [7629107] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629107/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629107/ reached maximum tries (51) after waiting for 300 seconds [7629108] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:46:23 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629108/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629108/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4905ab564e4b4074bd94e9b52c1da1df timeout expired in wait_until_healthy [7629109] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:44:42 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629109/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629109/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce496267b13044078c4525f81cede65b timeout expired in wait_until_healthy [7629110] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:45:32 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629110/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629110/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e59cb050fb0241f288e174f675f76ff9 timeout expired in wait_until_healthy [7629111] upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:41:46 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629111/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629111/ "1711686715.2098854 mon.a (mon.0) 183 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7629113] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:56:19 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629113/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629113/ "2024-03-29T04:50:00.000142+0000 mon.a (mon.0) 468 : cluster 3 [WRN] OSD_DOWN: 1 osds down" in cluster log [7629114] upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:41:59 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629114/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629114/ "1711687038.104081 mon.a (mon.0) 366 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7629115] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:42 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629115/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1107ec1b9bcb439e95f3f2adf3c3e0c5 Command failed (workunit test suites/fsstress.sh) on smithi134 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7629116] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 02:55:56 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629116/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629116/ "2024-03-29T04:50:00.000145+0000 mon.a (mon.0) 623 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log [7629117] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:32:39 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629117/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629117/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1bd90dc60aa49c8989feeffe4ae7e2e timeout expired in wait_until_healthy [7629119] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:55:16 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629119/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629119/ reached maximum tries (51) after waiting for 300 seconds [7629120] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 03:23:05 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629120/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629120/ "2024-03-29T04:50:00.000240+0000 mon.a (mon.0) 856 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log [7629121] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:52:42 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629121/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef80694eb64479cb4f9f341f9fbf018 timeout expired in wait_until_healthy [7629123] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 03:16:58 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629123/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629123/ "2024-03-29T06:30:00.000154+0000 mon.a (mon.0) 5753 : cluster 4 [ERR] OSD_SCRUB_ERRORS: 3 scrub errors" in cluster log [7629125] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:26:22 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629125/ Command failed on smithi003 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.1d deep-scrub' [7629126] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:17 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629126/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629126/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b933f54bdefc4ba58978f7b820729c06 Command failed on smithi067 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fe206bda-ed85-11ee-95d0-87774f69a715 -e sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba -- bash -c 'ceph orch ps'" [7629127] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 02:45:06 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629127/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629127/ "2024-03-29T05:10:00.000191+0000 mon.a (mon.0) 1714 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log Dead ================================================================= [7629100] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629100/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629100/ Error reimaging machines: Expected smithi100's OS to be centos 8 but found centos 9 [7629124] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629124/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629124/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7629128] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629128/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629128/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7629090] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:37 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629090/ [7629091] upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} time: 00:09:07 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629091/ [7629096] upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:11:11 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629096/ [7629112] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:37:56 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629112/ [7629118] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:29 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629118/ [7629122] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/iozone}} time: 00:16:32 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629122/ [7629129] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} time: 01:34:30 info: https://pulpito.ceph.com/teuthology-2024-03-29_02:08:11-upgrade-squid-distro-default-smithi/7629129/