2022-06-17T02:11:04.827 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T02:11:04.828 INFO:teuthology.results:subset: '25/32' 2022-06-17T02:11:04.828 INFO:teuthology.results:seed: '657' 2022-06-17T02:12:08.658 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T02:12:08.661 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-17T03:33:14.831 INFO:teuthology.results:Tests finished! gathering results... 2022-06-17T03:33:18.398 INFO:scrape:Found 43 jobs 2022-06-17T03:33:30.313 INFO:scrape:Found 5 distinct failure reasons 2022-06-17T03:33:30.314 INFO:scrape:Failure: Command failed on smithi078 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' 2022-06-17T03:33:30.314 INFO:scrape:34 jobs 2022-06-17T03:33:30.315 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2022-06-17T03:33:30.316 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'roles', 'scrub/no', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'syntax}', 'tasks/{0-check-counter', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/fs/norstats}}', 'workunit/kernel_untar_build}}', 'workunit/suites/dbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/iogen}}', 'workunit/suites/pjd}}', 'wsync/no}', 'wsync/yes}'] 2022-06-17T03:33:30.316 INFO:scrape: 2022-06-17T03:33:30.317 INFO:scrape:Failure: "2022-06-17T02:41:12.904401+0000 mon.a (mon.0) 373 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-06-17T03:33:30.317 INFO:scrape:2 jobs: ['6883388', '6883348'] 2022-06-17T03:33:30.318 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2022-06-17T03:33:30.318 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2022-06-17T03:33:30.319 INFO:scrape: 2022-06-17T03:33:30.319 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-06-17T03:33:30.320 INFO:scrape:1 jobs: ['6883350'] 2022-06-17T03:33:30.320 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2022-06-17T03:33:30.321 INFO:scrape: 2022-06-17T03:33:30.321 INFO:scrape:Failure: Command failed on smithi114 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' 2022-06-17T03:33:30.321 INFO:scrape:1 jobs: ['6883372'] 2022-06-17T03:33:30.322 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-06-17T03:33:30.322 INFO:scrape: 2022-06-17T03:33:30.323 INFO:scrape:Failure: Test failure: test_subvolume_group_quota_legacy_subvolume_removal (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2022-06-17T03:33:30.323 INFO:scrape:1 jobs: ['6883364'] 2022-06-17T03:33:30.324 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2022-06-17T03:33:30.324 INFO:scrape: 2022-06-17T03:33:30.327 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/ failed: 39 dead: 0 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [6883348] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:57:44 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883348/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883348/ "2022-06-17T02:55:07.392004+0000 mon.a (mon.0) 492 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6883349] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:35 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883349/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cdf402c49a7457bbd1891630237796a Command failed on smithi043 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883350] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:23:22 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883350/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883350/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2c8f2af157b4c60bc616fae168dfd56 reached maximum tries (90) after waiting for 540 seconds [6883352] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:16:24 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883352/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da91fe10aeb14af2ba19310ac09aac76 Command failed on smithi102 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883353] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:30:22 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883353/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7a4567fbc9f45138a34f6ef1b3c7765 Command failed on smithi041 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883354] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:33:12 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883354/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4541e77899094b62a357beceeda1261a Command failed on smithi031 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883355] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:19:56 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883355/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17cfc1d0e0c94ea387cdfc176c05d28b Command failed on smithi109 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883356] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:14:25 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883356/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8613e0a0710b43f88556942f87b90199 Command failed on smithi061 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883357] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:30:45 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883357/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a6b1bfe3ba541a5b70fe1f9eb8e56b3 Command failed on smithi001 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883358] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:14:19 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883358/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdb3588a8d3845ddb9d3d124db1e2fb8 Command failed on smithi078 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883360] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:20:15 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883360/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4aa654e679f8409784af31c9a5c99f1a Command failed on smithi037 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883361] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:34:05 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883361/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80cd7bdfede5423ab2a5da84a25d1635 Command failed on smithi160 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883362] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883362/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=858dca9b63054747bbbd3b6c3e165524 Command failed on smithi039 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883363] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:24:09 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883363/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc293a13f2cb4d848e0d910a2cf89036 Command failed on smithi119 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883364] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:32:38 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883364/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0f43e1eaf3547fbb06ba1471f3a8372 Test failure: test_subvolume_group_quota_legacy_subvolume_removal (tasks.cephfs.test_volumes.TestSubvolumeGroups) [6883366] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:14:50 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883366/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7ad2c7045314e2e9f20860614dc5432 Command failed on smithi057 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883367] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:29:13 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883367/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81bccd12cc6a45a4afbc04ba89553fd2 Command failed on smithi096 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883368] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:22:34 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883368/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ede246eb0e8540b28d014c051b56f548 Command failed on smithi101 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883369] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:14:55 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883369/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=931a0e0080db462d81d0b033eb65a7b2 Command failed on smithi063 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883370] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:33:01 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883370/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=260aa93e741c42319eb76a4761039041 Command failed on smithi008 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883371] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:16:31 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883371/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a824ff6a63eb4b7b9d41a8b19ae961aa Command failed on smithi085 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883372] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:19:18 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883372/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c74006b61d864ccda156f35c653d4e1b Command failed on smithi114 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883373] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:31:24 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883373/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14c493d979c4489eb21ee00631f27a88 Command failed on smithi139 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883374] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:19:54 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883374/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35b90c1d669e4a8087b8e9835ee49b10 Command failed on smithi136 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883375] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:32:07 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883375/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883375/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59585db373d14b9c8e4730d51d61150f Command failed on smithi052 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883377] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:24:50 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883377/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e336e4643b8a418bae6601c8c6ab828d Command failed on smithi036 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883378] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:33:57 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883378/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb95574b7ccd476f9ea5ab5fedcb8597 Command failed on smithi027 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883379] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:25:00 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883379/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db88ef461a634fcbacd5546eb3525f1d Command failed on smithi022 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883380] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:15:15 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883380/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca5b1c7c093e474ba67046c19b800448 Command failed on smithi156 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883381] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:20:13 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883381/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2eabe62fbfb442629b22351f9c9afcd6 Command failed on smithi071 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883382] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:17:29 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883382/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22ff3038477d4b31b3030ee251bb934c Command failed on smithi088 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883383] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:23:26 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883383/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffe210d6186a404c823776bef2c5aded Command failed on smithi066 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883384] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:34:54 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883384/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883384/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e02db4183dc4820bc7a5ebd5c146c1e Command failed on smithi007 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883385] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:18:13 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883385/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa14318bd96a44d5836da1a7875b75c6 Command failed on smithi019 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883386] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:16:56 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883386/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0238481f98b94720a844f86ee640dbb1 Command failed on smithi138 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:18:04 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883387/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c835971fcfa743acbdfec161f3c2eead Command failed on smithi003 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883388] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:52:24 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883388/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883388/ "2022-06-17T02:41:12.904401+0000 mon.a (mon.0) 373 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6883389] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:20:54 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883389/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c866a55a45945c1a1eb6e1569a44c6d Command failed on smithi073 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' [6883390] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:20:05 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883390/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e475a43f0ade4ecc904ccef6f8c6158c Command failed on smithi104 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git@github.com:vshankar/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a426c23a0727efc9b5a05e3ad12e6bcf24b901bc' Pass ================================================================= [6883351] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:58:20 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883351/ [6883359] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:20:36 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883359/ [6883365] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:43:25 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883365/ [6883376] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:22 info: http://pulpito.ceph.com/vshankar-2022-06-17_02:10:55-fs-wip-vshankar-testing1-20220615-072516-testing-default-smithi/6883376/