2024-02-23T10:48:24.311 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-23T10:48:24.311 INFO:teuthology.results:seed: '639' 2024-02-23T12:03:03.292 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-23T12:03:03.317 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-23T22:59:52.215 INFO:teuthology.results:Tests finished! gathering results... 2024-02-23T23:00:00.524 INFO:scrape:Found 90 jobs 2024-02-23T23:00:31.735 INFO:scrape:Found 11 distinct failure reasons 2024-02-23T23:00:31.735 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-02-23T23:00:31.735 INFO:scrape:19 jobs: ['7571877', '7571843', '7571839', '7571847', '7571861', '7571875', '7571888', '7571845', '7571893', '7571882', '7571926', '7571879', '7571904', '7571862', '7571897', '7571890', '7571885', '7571900', '7571852'] 2024-02-23T23:00:31.736 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-02-23T23:00:31.736 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-02-23T23:00:31.736 INFO:scrape: 2024-02-23T23:00:31.736 INFO:scrape:Failure: Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45794ba4-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' 2024-02-23T23:00:31.736 INFO:scrape:11 jobs: ['7571846', '7571876', '7571842', '7571871', '7571859', '7571855', '7571878', '7571868', '7571880', '7571894', '7571865'] 2024-02-23T23:00:31.737 INFO:scrape:suites intersection: ['1-volume/{0-create', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-02-23T23:00:31.737 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-02-23T23:00:31.737 INFO:scrape: 2024-02-23T23:00:31.737 INFO:scrape:Failure: No module named 'tasks.workunit' 2024-02-23T23:00:31.737 INFO:scrape:9 jobs: ['7571911', '7571905', '7571907', '7571914', '7571906', '7571913', '7571912', '7571910', '7571927'] 2024-02-23T23:00:31.737 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-02-23T23:00:31.738 INFO:scrape:suites union: ['1-mount/mount/fuse', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-clients/fuse-upgrade', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '3-workload/stress_tests/fsstress}}', '3-workload/stress_tests/kernel_untar_build}}', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'branch/nautilus', 'centos_8.stream', 'centos_9.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs:upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'tasks/{0-install'] 2024-02-23T23:00:31.738 INFO:scrape: 2024-02-23T23:00:31.738 INFO:scrape:Failure: Command failed on smithi161 with status 1: 'sudo yum -y install cephfs-java' 2024-02-23T23:00:31.738 INFO:scrape:10 jobs: ['7571925', '7571915', '7571923', '7571922', '7571916', '7571921', '7571919', '7571924', '7571920', '7571918'] 2024-02-23T23:00:31.738 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-02-23T23:00:31.738 INFO:scrape:suites union: ['1-client', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '2-upgrade', '3-compat_client/quincy}}', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}', 'tasks/{0-octopus'] 2024-02-23T23:00:31.738 INFO:scrape: 2024-02-23T23:00:31.738 INFO:scrape:Failure: "1708688400.0001826 mon.smithi078 (mon.0) 667 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log 2024-02-23T23:00:31.738 INFO:scrape:6 jobs: ['7571898', '7571908', '7571863', '7571873', '7571869', '7571867'] 2024-02-23T23:00:31.739 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-23T23:00:31.739 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-23T23:00:31.739 INFO:scrape: 2024-02-23T23:00:31.739 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi077.front.sepia.ceph.com: ['type=AVC msg=audit(1708686890.477:210): avc: denied { checkpoint_restore } for pid=1204 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1', 'type=AVC msg=audit(1708686890.477:211): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-23T23:00:31.739 INFO:scrape:3 jobs: ['7571860', '7571857', '7571874'] 2024-02-23T23:00:31.739 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-02-23T23:00:31.739 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.0}'] 2024-02-23T23:00:31.739 INFO:scrape: 2024-02-23T23:00:31.739 INFO:scrape:Failure: Command failed on smithi157 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' 2024-02-23T23:00:31.740 INFO:scrape:1 jobs: ['7571917'] 2024-02-23T23:00:31.740 INFO:scrape:suites: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/blogbench}}', 'branch/nautilus', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs:upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-install'] 2024-02-23T23:00:31.740 INFO:scrape: 2024-02-23T23:00:31.740 INFO:scrape:Dead: b'2024-02-23T22:56:49.680 DEBUG:teuthology.exit:Finished running handlers' 2024-02-23T23:00:31.740 INFO:scrape:1 jobs: ['7571849'] 2024-02-23T23:00:31.740 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-02-23T23:00:31.740 INFO:scrape: 2024-02-23T23:00:31.740 INFO:scrape:Failure: "2024-02-23T11:40:00.000248+0000 mon.smithi102 (mon.0) 585 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 1490/375 objects degraded (397.333%), 2 pgs degraded" in cluster log 2024-02-23T23:00:31.740 INFO:scrape:2 jobs: ['7571887', '7571848'] 2024-02-23T23:00:31.740 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-02-23T23:00:31.741 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-02-23T23:00:31.741 INFO:scrape: 2024-02-23T23:00:31.741 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-23T23:00:31.741 INFO:scrape:1 jobs: ['7571844'] 2024-02-23T23:00:31.741 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-02-23T23:00:31.741 INFO:scrape: 2024-02-23T23:00:31.741 INFO:scrape:Failure: Command failed on smithi114 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub' 2024-02-23T23:00:31.741 INFO:scrape:1 jobs: ['7571870'] 2024-02-23T23:00:31.741 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2024-02-23T23:00:31.741 INFO:scrape: 2024-02-23T23:00:31.748 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/ failed: 63 dead: 1 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [7571839] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:51:40 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571839/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571839/ reached maximum tries (51) after waiting for 300 seconds [7571842] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:13:46 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571842/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29d31286963a439e8bd47faa8d694227 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80e2d926-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' [7571843] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:41 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571843/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571843/ reached maximum tries (51) after waiting for 300 seconds [7571844] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 04:15:32 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571844/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571844/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6089a6bada51421cbceaa780060ae351 Command failed (workunit test suites/fsstress.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7571845] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:49:26 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571845/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571845/ reached maximum tries (51) after waiting for 300 seconds [7571846] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:31 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571846/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2e40aec66454c9fa3efed73137357b9 Command failed on smithi019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45794ba4-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7571847] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:34 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571847/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571847/ reached maximum tries (51) after waiting for 300 seconds [7571848] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:45 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571848/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571848/ "2024-02-23T11:30:00.000246+0000 mon.smithi007 (mon.0) 239 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 43/219 objects degraded (19.635%), 17 pgs degraded" in cluster log [7571852] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:18:41 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571852/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571852/ reached maximum tries (51) after waiting for 300 seconds [7571855] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:55:14 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571855/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=421963d2de364342aef7d7d8622df473 Command failed on smithi149 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 58af3530-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7571857] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:43:55 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571857/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571857/ SELinux denials found on ubuntu@smithi117.front.sepia.ceph.com: ['type=AVC msg=audit(1708686913.325:207): avc: denied { checkpoint_restore } for pid=1151 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7571859] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:56:25 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571859/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac161a6d6a3f4f6a973205a1154db131 Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 81ca7380-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 1\'"\'"\'\'' [7571860] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:58:16 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571860/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571860/ SELinux denials found on ubuntu@smithi077.front.sepia.ceph.com: ['type=AVC msg=audit(1708686890.477:210): avc: denied { checkpoint_restore } for pid=1204 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1', 'type=AVC msg=audit(1708686890.477:211): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7571861] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:49:16 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571861/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571861/ reached maximum tries (51) after waiting for 300 seconds [7571862] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:41 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571862/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571862/ reached maximum tries (51) after waiting for 300 seconds [7571863] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:43 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571863/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571863/ "1708688400.0004654 mon.smithi137 (mon.0) 759 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log [7571865] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:33:09 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571865/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1de9c20dfc5e46aaa847d2dc0cd17328 Command failed on smithi072 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ba22e3c-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571867] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:38 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571867/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571867/ "1708688400.0001185 mon.smithi003 (mon.0) 795 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log [7571868] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:19:32 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571868/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4bb1df99bd04270887c981c2abf11f2 Command failed on smithi089 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e34dbfd2-d23d-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571869] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:49 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571869/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571869/ "1708688400.0001245 mon.smithi038 (mon.0) 962 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log [7571870] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:26:48 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571870/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571870/ Command failed on smithi114 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub' [7571871] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:25:41 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571871/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af3060909212404dba577d1e482ba9ec Command failed on smithi174 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a7248e-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571873] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:48:10 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571873/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571873/ "1708687800.0001059 mon.smithi006 (mon.0) 880 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log [7571874] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:42:14 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571874/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571874/ SELinux denials found on ubuntu@smithi067.front.sepia.ceph.com: ['type=AVC msg=audit(1708686936.630:210): avc: denied { checkpoint_restore } for pid=1187 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7571875] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:19:57 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571875/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571875/ reached maximum tries (51) after waiting for 300 seconds [7571876] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:33:36 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571876/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7df41343ac0d47dab47096d856ce2a40 Command failed on smithi039 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fbe89f04-d23c-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571877] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:34:12 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571877/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571877/ reached maximum tries (51) after waiting for 300 seconds [7571878] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:21:42 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571878/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dc4c24bebf748b19973123b91b79565 Command failed on smithi045 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e84df6e-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571879] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:51:57 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571879/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571879/ reached maximum tries (51) after waiting for 300 seconds [7571880] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:19:14 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571880/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dcd03cf11f964f4984baf4e2a837890f Command failed on smithi135 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3a880992-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571882] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:52:58 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571882/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571882/ reached maximum tries (51) after waiting for 300 seconds [7571885] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:57:07 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571885/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571885/ reached maximum tries (51) after waiting for 300 seconds [7571887] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:26 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571887/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571887/ "2024-02-23T11:40:00.000248+0000 mon.smithi102 (mon.0) 585 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 1490/375 objects degraded (397.333%), 2 pgs degraded" in cluster log [7571888] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:49:29 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571888/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571888/ reached maximum tries (51) after waiting for 300 seconds [7571890] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:36:07 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571890/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571890/ reached maximum tries (51) after waiting for 300 seconds [7571893] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:22:43 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571893/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571893/ reached maximum tries (51) after waiting for 300 seconds [7571894] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:24:19 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571894/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=071dda348be043b8bc1dbcb8b9655f4d Command failed on smithi084 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70a1f272-d23e-11ee-95c0-87774f69a715 -e sha1=1eb3719c3f0a975bd84920c85c7db3ed9c91b99d -- bash -c 'ceph orch ps'" [7571897] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:13 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571897/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571897/ reached maximum tries (51) after waiting for 300 seconds [7571898] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:56 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571898/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571898/ "1708688400.0001826 mon.smithi078 (mon.0) 667 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log [7571900] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:45:55 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571900/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571900/ reached maximum tries (51) after waiting for 300 seconds [7571904] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:44:40 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571904/ reached maximum tries (51) after waiting for 300 seconds [7571905] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:22:14 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571905/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571905/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9cfc45ab9f9b40a7a9fcd2f1a9f02122 No module named 'tasks.fs' [7571906] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:22:28 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571906/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571906/ sentry: https://sentry.ceph.com/organizations/ceph/?query=542a50203e30459f9c74084b602fd73b No module named 'tasks.fs' [7571907] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:22:45 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571907/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e520ba755bbd494ea058449bef62edc5 No module named 'tasks.fs' [7571908] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:41:30 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571908/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571908/ "1708690200.0001154 mon.smithi120 (mon.0) 785 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data" in cluster log [7571910] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:21:08 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571910/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=300a3288fe314393a19d7cc5a9efaf9c No module named 'tasks.fs' [7571911] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} ----------------------------------------------------------------- time: 00:14:57 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571911/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22852b7b43e648fe83debd009253f29f No module named 'tasks.workunit' [7571912] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:36 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571912/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aea069a2d54f4dfc8fbbf964d4c29ec6 No module named 'tasks.cephadm' [7571913] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:11:23 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571913/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=228e8cb9feae42d1ad8f559395030531 No module named 'tasks.cephadm' [7571914] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:19 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571914/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571914/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb562c6ac9824992bcdf03217e477b0f No module named 'tasks.cephadm' [7571915] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:09:07 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571915/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f929bfcee4e464086a4a7cc3cc8b63d Command failed on smithi178 with status 1: 'sudo yum -y install cephfs- java' [7571916] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:03 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571916/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=846fa66f59db44cdb9553545b001dcf1 Command failed on smithi131 with status 1: 'sudo yum -y install cephfs- java' [7571917] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} ----------------------------------------------------------------- time: 00:07:51 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571917/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49db83ac19a54007898dceba15ed0d1c Command failed on smithi157 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl- devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7571918] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:09 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571918/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3db7ca555e0e4db39a014d3bed3e6932 Command failed on smithi115 with status 1: 'sudo yum -y install cephfs- java' [7571919] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:11 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571919/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571919/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec85c6e1cf514672ba1c23c4a44a7ffd Command failed on smithi007 with status 1: 'sudo yum -y install cephfs- java' [7571920] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:15 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571920/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2adc0a969e514efe8725b4693e8d75d6 Command failed on smithi184 with status 1: 'sudo yum -y install cephfs- java' [7571921] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:03 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571921/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf75d85b53b8420587193422415623bf Command failed on smithi114 with status 1: 'sudo yum -y install cephfs- java' [7571922] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:04 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571922/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fa36375559145d9b5bb9ebaabd3e4d5 Command failed on smithi012 with status 1: 'sudo yum -y install cephfs- java' [7571923] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:26 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571923/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571923/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d32ab760f57949a58077d7d54b9819b9 Command failed on smithi170 with status 1: 'sudo yum -y install cephfs- java' [7571924] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:08:02 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571924/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=562d11d15109400c83d57a08ed50886d Command failed on smithi152 with status 1: 'sudo yum -y install cephfs- java' [7571925] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:07:43 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571925/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571925/ sentry: https://sentry.ceph.com/organizations/ceph/?query=efba753c98984211a7b408fc215b3b16 Command failed on smithi161 with status 1: 'sudo yum -y install cephfs- java' [7571926] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:37 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571926/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571926/ reached maximum tries (51) after waiting for 300 seconds [7571927] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} ----------------------------------------------------------------- time: 00:09:23 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571927/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=579c31ee23b74b47b603c8fa053ff5dc No module named 'tasks' Dead ================================================================= [7571849] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571849/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571849/ hit max job timeout Pass ================================================================= [7571838] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:18 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571838/ [7571840] fs:upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:15:21 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571840/ [7571841] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} time: 00:15:10 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571841/ [7571850] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}} time: 00:17:55 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571850/ [7571851] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} time: 00:21:13 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571851/ [7571853] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:44:43 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571853/ [7571854] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:24:43 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571854/ [7571856] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} time: 00:27:14 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571856/ [7571858] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:54:36 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571858/ [7571864] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:53:48 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571864/ [7571866] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/fsstress}} time: 00:21:21 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571866/ [7571872] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/blogbench}} time: 00:26:15 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571872/ [7571881] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/iozone}} time: 00:17:44 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571881/ [7571883] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} time: 01:52:50 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571883/ [7571884] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:12 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571884/ [7571886] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} time: 00:15:50 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571886/ [7571889] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:43:57 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571889/ [7571891] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} time: 00:39:48 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571891/ [7571892] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:31 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571892/ [7571895] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:45:55 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571895/ [7571896] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} time: 00:20:39 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571896/ [7571899] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:29:06 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571899/ [7571901] fs:upgrade/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} time: 00:28:26 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571901/ [7571902] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:46:30 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571902/ [7571903] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} time: 00:52:49 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571903/ [7571909] fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/dbench}} time: 00:51:45 info: https://pulpito.ceph.com/vshankar-2024-02-23_10:48:16-fs:upgrade-wip-64502-testing-default-smithi/7571909/