2024-04-13T06:44:04.095 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-13T06:44:04.096 INFO:teuthology.results:subset: '6/32' 2024-04-13T06:44:04.096 INFO:teuthology.results:seed: '589' 2024-04-13T08:17:53.680 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-13T08:17:53.706 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-13T11:21:08.455 INFO:teuthology.results:Tests finished! gathering results... 2024-04-13T11:21:11.447 INFO:scrape:Found 37 jobs 2024-04-13T11:21:47.865 INFO:scrape:Found 8 distinct failure reasons 2024-04-13T11:21:47.866 INFO:scrape:Failure: "1712992412.8671365 mon.a (mon.0) 502 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2024-04-13T11:21:47.866 INFO:scrape:12 jobs: ['7654937', '7654964', '7654949', '7654931', '7654951', '7654954', '7654930', '7654945', '7654958', '7654934', '7654953', '7654950'] 2024-04-13T11:21:47.866 INFO:scrape:suites intersection: [] 2024-04-13T11:21:47.867 INFO:scrape:suites union: ['-', '0-roles', '0-start', '1-ceph-install/quincy', '1-ranks/1', '1-start', '1-tasks', '1-tasks}', '1-volume/{0-create', '2', '2-allow_standby_replay/no', '2-client/kclient', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-inline/no', '3-stress-tasks/{radosbench', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/filestore-xfs', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'roles', 'snaps-few-objects}', 'start}', 'syntax}', 'tasks/{0-from/quincy', 'test_rbd_api', 'test_rbd_python}}', 'ubuntu_latest}', 'upgrade', 'upgrade-sequence', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap', 'upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default'] 2024-04-13T11:21:47.867 INFO:scrape: 2024-04-13T11:21:47.867 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-13T11:21:47.867 INFO:scrape:3 jobs: ['7654946', '7654963', '7654933'] 2024-04-13T11:21:47.867 INFO:scrape:suites intersection: ['1-ranks/2', '1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-04-13T11:21:47.867 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-04-13T11:21:47.867 INFO:scrape: 2024-04-13T11:21:47.867 INFO:scrape:Failure: "2024-04-13T09:47:53.272579+0000 osd.4 (osd.4) 47 : cluster [ERR] 58.13 soid 58:c93822fb:::smithi005677167-404:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log 2024-04-13T11:21:47.868 INFO:scrape:9 jobs: ['7654960', '7654943', '7654947', '7654959', '7654935', '7654961', '7654939', '7654965', '7654938'] 2024-04-13T11:21:47.868 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2024-04-13T11:21:47.868 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc'] 2024-04-13T11:21:47.868 INFO:scrape: 2024-04-13T11:21:47.868 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific 2024-04-13T11:21:47.868 INFO:scrape:1 jobs: ['7654932'] 2024-04-13T11:21:47.868 INFO:scrape:suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream}'] 2024-04-13T11:21:47.868 INFO:scrape: 2024-04-13T11:21:47.868 INFO:scrape:Failure: "2024-04-13T08:04:05.829364+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log 2024-04-13T11:21:47.868 INFO:scrape:1 jobs: ['7654957'] 2024-04-13T11:21:47.869 INFO:scrape:suites: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-04-13T11:21:47.869 INFO:scrape: 2024-04-13T11:21:47.869 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-13T11:21:47.869 INFO:scrape:2 jobs: ['7654940', '7654952'] 2024-04-13T11:21:47.869 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-04-13T11:21:47.869 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap'] 2024-04-13T11:21:47.869 INFO:scrape: 2024-04-13T11:21:47.869 INFO:scrape:Failure: timeout expired in wait_until_healthy 2024-04-13T11:21:47.869 INFO:scrape:3 jobs: ['7654941', '7654942', '7654955'] 2024-04-13T11:21:47.869 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] 2024-04-13T11:21:47.870 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04'] 2024-04-13T11:21:47.870 INFO:scrape: 2024-04-13T11:21:47.870 INFO:scrape:Failure: Command failed on smithi017 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub' 2024-04-13T11:21:47.870 INFO:scrape:1 jobs: ['7654956'] 2024-04-13T11:21:47.870 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap'] 2024-04-13T11:21:47.870 INFO:scrape: 2024-04-13T11:21:47.873 INFO:teuthology.results:Sending results to ceph-qa@ceph.com: Test Run: teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/ failed: 32 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7654930] upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} ----------------------------------------------------------------- time: 00:09:51 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654930/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654930/ "2024-04-13T07:05:44.150383+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7654931] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:49:34 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654931/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654931/ "2024-04-13T07:18:14.755811+0000 mon.a (mon.0) 348 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7654932] upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:04:43 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654932/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b55d66f184a413ca891fef6bf11d5ab Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=paci fic [7654933] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:50 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654933/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654933/ reached maximum tries (51) after waiting for 300 seconds [7654934] upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:43:30 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654934/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654934/ "2024-04-13T07:26:53.856639+0000 mon.a (mon.0) 18 : cluster [ERR] Health check failed: 8 osds(s) are not reachable (OSD_UNREACHABLE)" in cluster log [7654935] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:54:43 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654935/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654935/ "2024-04-13T08:58:31.109022+0000 osd.6 (osd.6) 4 : cluster [ERR] 62.2 soid 62:431a08bb:::smithi162701712-19:18b : object info inconsistent , attr value mismatch '__header'" in cluster log [7654937] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:47:57 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654937/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654937/ "1712992412.8671365 mon.a (mon.0) 502 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7654938] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:01:25 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654938/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654938/ "2024-04-13T08:38:45.109328+0000 osd.3 (osd.3) 72 : cluster [ERR] 58.3 soid 58:c1a65454:::smithi161761142-153:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654939] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 02:42:50 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654939/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654939/ "2024-04-13T08:23:17.160968+0000 osd.5 (osd.5) 33 : cluster [ERR] 50.17 soid 50:e9195193:::smithi044512902-359:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654940] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:35:59 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654940/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654940/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9cfeee000a094fd88d662b582a904478 Command failed (workunit test suites/fsstress.sh) on smithi130 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c2d8c3155d854ee204cb7c7261232571229d1d54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7654941] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:44:31 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654941/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddee5c7d8976497daf3b06d0f67fa21e timeout expired in wait_until_healthy [7654942] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 00:45:23 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654942/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6399b7036af425da38bc9e9ae318644 timeout expired in wait_until_healthy [7654943] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:38:31 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654943/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654943/ "2024-04-13T09:28:17.435283+0000 osd.5 (osd.5) 50 : cluster [ERR] 103.0 soid 103:002a3059:::smithi053842980-10:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654945] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 02:40:59 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654945/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654945/ "1712993619.5137918 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7654946] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:07:26 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654946/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654946/ reached maximum tries (51) after waiting for 300 seconds [7654947] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:29:22 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654947/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654947/ "2024-04-13T09:00:15.366179+0000 osd.4 (osd.4) 72 : cluster [ERR] 58.6 soid 58:604823a0:::smithi090704779-483:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654949] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:20:23 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654949/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654949/ "1712993807.007602 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7654950] upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} ----------------------------------------------------------------- time: 00:30:12 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654950/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654950/ "1712994145.6915643 mon.a (mon.0) 660 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7654951] upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:47:28 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654951/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654951/ "2024-04-13T07:50:01.317903+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7654952] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:36:38 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654952/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f482098f55974e25b0c008c4f8e861b4 Command failed (workunit test suites/fsstress.sh) on smithi077 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c2d8c3155d854ee204cb7c7261232571229d1d54 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7654953] upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 01:28:46 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654953/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654953/ "1712994291.1762464 mon.a (mon.0) 525 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7654954] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:25:37 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654954/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654954/ "2024-04-13T08:18:01.228290+0000 mon.a (mon.0) 176 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [7654955] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 00:50:38 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654955/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654955/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e59bbb282b54a73a7917c1242012144 timeout expired in wait_until_healthy [7654956] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:21:56 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654956/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654956/ Command failed on smithi017 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub' [7654957] upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 03:16:30 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654957/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654957/ "2024-04-13T08:04:05.829364+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log [7654958] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:34:01 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654958/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654958/ "1712995535.7198308 mon.smithi042 (mon.0) 341 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log [7654959] upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:58:39 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654959/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654959/ "2024-04-13T10:05:53.274209+0000 osd.1 (osd.1) 86 : cluster [ERR] 53.1 soid 53:810a7c1d:::smithi098379425-22:19f : object info inconsistent , attr value mismatch '__header'" in cluster log [7654960] upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 03:08:57 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654960/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654960/ "2024-04-13T09:47:53.272579+0000 osd.4 (osd.4) 47 : cluster [ERR] 58.13 soid 58:c93822fb:::smithi005677167-404:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654961] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} ----------------------------------------------------------------- time: 02:42:32 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654961/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654961/ "2024-04-13T09:33:49.264953+0000 osd.6 (osd.6) 153 : cluster [ERR] 50.19 soid 50:9882d297:::smithi089529128-37:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log [7654963] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:39 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654963/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654963/ reached maximum tries (51) after waiting for 300 seconds [7654964] upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:31:30 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654964/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654964/ "2024-04-13T08:48:43.355620+0000 mon.a (mon.0) 268 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [7654965] upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} ----------------------------------------------------------------- time: 02:19:42 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654965/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654965/ "2024-04-13T09:47:42.801519+0000 osd.6 (osd.6) 34 : cluster [ERR] 50.10 soid 50:096cad85:::smithi049548649-6:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log Pass ================================================================= [7654929] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:24:13 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654929/ [7654936] upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} time: 00:11:49 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654936/ [7654944] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:20:28 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654944/ [7654948] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:18:43 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654948/ [7654962] upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:23:02 info: https://pulpito.ceph.com/teuthology-2024-04-13_03:08:05-upgrade-main-distro-default-smithi/7654962/