2022-10-12T17:34:08.987 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T17:34:08.988 INFO:teuthology.results:subset: '24/64' 2022-10-12T17:34:08.989 INFO:teuthology.results:seed: '9561' 2022-10-12T19:28:21.180 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-12T19:28:21.207 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-13T07:05:07.939 INFO:teuthology.results:Tests finished! gathering results... 2022-10-13T07:05:11.475 INFO:scrape:Found 36 jobs 2022-10-13T07:05:29.838 INFO:scrape:Found 11 distinct failure reasons 2022-10-13T07:05:29.839 INFO:scrape:Dead: b'2022-10-13T06:58:28.197 DEBUG:teuthology.exit:Finished running handlers' 2022-10-13T07:05:29.840 INFO:scrape:1 jobs: ['7064206'] 2022-10-13T07:05:29.840 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-10-13T07:05:29.841 INFO:scrape: 2022-10-13T07:05:29.841 INFO:scrape:Failure: Command failed on smithi036 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25ca36ac-4a58-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" 2022-10-13T07:05:29.842 INFO:scrape:5 jobs: ['7064190', '7064195', '7064208', '7064200', '7064215'] 2022-10-13T07:05:29.842 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2022-10-13T07:05:29.843 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2022-10-13T07:05:29.843 INFO:scrape: 2022-10-13T07:05:29.844 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2022-10-13T07:05:29.844 INFO:scrape:2 jobs: ['7064216', '7064191'] 2022-10-13T07:05:29.845 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2022-10-13T07:05:29.845 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2022-10-13T07:05:29.846 INFO:scrape: 2022-10-13T07:05:29.846 INFO:scrape:Failure: Test failure: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2022-10-13T07:05:29.846 INFO:scrape:1 jobs: ['7064199'] 2022-10-13T07:05:29.847 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2022-10-13T07:05:29.847 INFO:scrape: 2022-10-13T07:05:29.848 INFO:scrape:Failure: "1665600116.6346805 mon.a (mon.0) 181 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log 2022-10-13T07:05:29.848 INFO:scrape:4 jobs: ['7064197', '7064214', '7064185', '7064207'] 2022-10-13T07:05:29.849 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2022-10-13T07:05:29.849 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/mixed-clients/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/workunit/snaps}'] 2022-10-13T07:05:29.850 INFO:scrape: 2022-10-13T07:05:29.850 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2022-10-13T07:05:29.851 INFO:scrape:2 jobs: ['7064203', '7064189'] 2022-10-13T07:05:29.851 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-10-13T07:05:29.851 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-10-13T07:05:29.852 INFO:scrape: 2022-10-13T07:05:29.852 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2022-10-13T07:05:29.853 INFO:scrape:12 jobs: ['7064212', '7064193', '7064205', '7064217', '7064198', '7064186', '7064183', '7064192', '7064196', '7064202', '7064210', '7064211'] 2022-10-13T07:05:29.853 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2022-10-13T07:05:29.854 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2022-10-13T07:05:29.854 INFO:scrape: 2022-10-13T07:05:29.855 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2022-10-13T07:05:29.855 INFO:scrape:1 jobs: ['7064213'] 2022-10-13T07:05:29.856 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2022-10-13T07:05:29.856 INFO:scrape: 2022-10-13T07:05:29.857 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2022-10-13T07:05:29.857 INFO:scrape:1 jobs: ['7064188'] 2022-10-13T07:05:29.857 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] 2022-10-13T07:05:29.858 INFO:scrape: 2022-10-13T07:05:29.858 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-10-13T07:05:29.859 INFO:scrape:1 jobs: ['7064187'] 2022-10-13T07:05:29.859 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-10-13T07:05:29.860 INFO:scrape: 2022-10-13T07:05:29.860 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c69816286ae6ae1422f2701ef2872f593f4223da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-10-13T07:05:29.861 INFO:scrape:2 jobs: ['7064204', '7064209'] 2022-10-13T07:05:29.861 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount/fuse', 'osd}'] 2022-10-13T07:05:29.861 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-10-13T07:05:29.862 INFO:scrape: 2022-10-13T07:05:29.864 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/ failed: 31 dead: 1 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [7064183] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:24:58 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064183/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f19eb974c1440599105b481dd1ee963 Command failed on smithi102 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064185] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:26:48 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064185/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064185/ "1665598673.5803294 mon.a (mon.0) 331 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7064186] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:24:57 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064186/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e61dff6d1edb4ba9bfe114f56150a6d2 Command failed on smithi154 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064187] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:30:47 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064187/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064187/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b53c385e30d4c13ac7fcc6ac2c32c46 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7064188] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064188/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c203dac14ecd4efc9412f9b59265a553 Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [7064189] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:45:32 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064189/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=725836c47a0f498fbef82567edd414c4 Command failed (workunit test suites/fsstress.sh) on smithi111 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c69816286ae6ae1422f2701ef2872f593f4223da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7064190] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:23 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064190/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=761c007f19174afba496beb1d00ef318 Command failed on smithi036 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25ca36ac-4a58-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7064191] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064191/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee9c3a7d2e584a669346ef73edc8724b Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [7064192] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:18:10 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064192/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=999195d5a8a1481497507306a69d39cf Command failed on smithi046 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064193] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:25:03 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064193/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac039bd086834fea992bd6716bbc1b6a Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064195] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:07 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=103583ba86f44f34a03190b143d1a81b Command failed on smithi036 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7a7ee7a-4a5b-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7064196] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:25:18 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064196/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccf2ee0b93b14d16b53b2f5e59a085a6 Command failed on smithi102 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064197] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:47:11 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064197/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064197/ "1665600116.6346805 mon.a (mon.0) 181 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7064198] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:18:38 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064198/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064198/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1181882669d4208ba794c127f8ba4f9 Command failed on smithi057 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:12:50 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064199/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=878b40c7e54747259739b296ca0e3b49 Test failure: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) [7064200] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:06 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064200/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9787d106cf55492e9b4c04e0c8f77c87 Command failed on smithi037 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 52b67124-4a5f-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7064202] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:24:56 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064202/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ede98a4a60324e718d065c5667fed2da Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064203] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:43:13 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064203/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1058de88f044191b23eea358a8124bb Command failed (workunit test suites/fsstress.sh) on smithi055 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c69816286ae6ae1422f2701ef2872f593f4223da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7064204] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:50:13 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064204/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9f035b11eeb47818430cb34decacf78 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c69816286ae6ae1422f2701ef2872f593f4223da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7064205] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:25:42 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064205/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3a98dc3fff14714bfdcafec75ba892c Command failed on smithi017 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064207] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 01:43:02 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064207/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064207/ "1665603230.716398 mon.a (mon.0) 361 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7064208] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064208/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b0cfecf298c4248ba180ce1611386d8 Command failed on smithi124 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0350e2-4a61-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7064209] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:44:48 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064209/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9106759f320402e928b0464105cdfd3 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c69816286ae6ae1422f2701ef2872f593f4223da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7064210] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:18:20 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064210/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064210/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa780fe87e774bf09345e8a4f06b2c2e Command failed on smithi037 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064211] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:25:24 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064211/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9dbcb4f462546b790fdb6ed5c558811 Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064212] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:25:29 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064212/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064212/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f72a7f5d84d4ab7b72408b5197f209a Command failed on smithi059 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7064213] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:28:28 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10fd667b81a1434795743f933da8b2dd Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) [7064214] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:17:44 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064214/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064214/ "1665603815.4542909 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 MDSs behind on trimming (MDS_TRIM)" in cluster log [7064215] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dddd3cb3d4d4b81a9df9ae052d8a53d Command failed on smithi124 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63934674-4a65-11ed-8437-001a4aab830c -e sha1=c69816286ae6ae1422f2701ef2872f593f4223da -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7064216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:12:10 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064216/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a68a300c82074e27b58713b65646625b Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [7064217] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:18:40 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064217/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb78e184617549f7a76533e56312d94b Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Dead ================================================================= [7064206] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064206/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064206/ hit max job timeout Pass ================================================================= [7064184] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:54 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064184/ [7064194] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:31:09 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064194/ [7064201] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:21 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064201/ [7064218] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:50:43 info: http://pulpito.ceph.com/vshankar-2022-10-12_17:33:04-fs-wip-vshankar-testing-20221011-145847-testing-default-smithi/7064218/