2023-07-20T16:50:33.866 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T16:50:33.885 INFO:teuthology.results:subset: '12/512' 2023-07-20T16:50:33.885 INFO:teuthology.results:seed: '8388' 2023-07-20T18:38:44.275 INFO:root:teuthology version: 0.0.1.dev153+g79491cb 2023-07-20T18:38:44.301 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-20T19:27:48.641 INFO:teuthology.results:Tests finished! gathering results... 2023-07-20T19:28:04.125 INFO:scrape:Found 97 jobs 2023-07-20T19:28:13.361 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345416/teuthology.log 2023-07-20T19:28:24.124 INFO:scrape:Found 3 distinct failure reasons 2023-07-20T19:28:24.125 INFO:scrape:Failure: Command failed on smithi149 with status 2: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' 2023-07-20T19:28:24.126 INFO:scrape:95 jobs 2023-07-20T19:28:24.128 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2023-07-20T19:28:24.128 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/distributed', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-07-20T19:28:24.129 INFO:scrape: 2023-07-20T19:28:24.129 INFO:scrape:Failure: Command failed on smithi135 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull' 2023-07-20T19:28:24.129 INFO:scrape:1 jobs: ['7345423'] 2023-07-20T19:28:24.130 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/direct_io}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2023-07-20T19:28:24.130 INFO:scrape: 2023-07-20T19:28:24.130 INFO:scrape:Dead: None 2023-07-20T19:28:24.131 INFO:scrape:1 jobs: ['7345416'] 2023-07-20T19:28:24.131 INFO:scrape: 2023-07-20T19:28:24.138 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/ failed: 96 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7345327] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:30:00 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345327/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7992d7ea417e4ab68499835e14f7646f Command failed on smithi016 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345328] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:28:10 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345328/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f75c63060c354474b55babb4a99b2bd6 Command failed on smithi037 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345329] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:29:37 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345329/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345329/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7006ba40b66748d0acd629c640b09534 Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345330] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:31:41 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345330/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345330/ sentry: https://sentry.ceph.com/organizations/ceph/?query=079dcf03a74e48dc9571ca4b2005511c Command failed on smithi069 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345331] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:23:08 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345331/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345331/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0aebd41d7174868bddf5614f6b8b410 Command failed on smithi062 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345332] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:28:40 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345332/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a140af45d6f44714b2d4e89974c872a5 Command failed on smithi006 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345333] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:27:44 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345333/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33b4c085e90f4ba2bce2b5988e8c125b Command failed on smithi064 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345334] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:29:04 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345334/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345334/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0919b66267849f1b12034fab84ebfac Command failed on smithi050 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345335] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:33:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345335/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d40582f7e844541b278f85a65f913aa Command failed on smithi067 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345336] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:33:35 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345336/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=725ee01972ac41f9b91f05ee56fba011 Command failed on smithi044 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345337] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:33:45 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9f35efce74b40bd98105c616b76f6d2 Command failed on smithi049 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345338] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:39:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345338/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e20e744442e47c3b1534f04127d9ea0 Command failed on smithi007 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345339] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:28:31 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345339/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345339/ sentry: https://sentry.ceph.com/organizations/ceph/?query=36ed4d1176d7492cbebd13636e9e817d Command failed on smithi040 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345340] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:35:14 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345340/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67008b76d440427c9e4138168d8ce823 Command failed on smithi019 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345341] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:28:33 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345341/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de2b8e10f44e406eb3da530578034e41 Command failed on smithi098 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345342] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:28:01 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345342/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8501bd27134d4e19918f4f8e57cfca93 Command failed on smithi112 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345343] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:27:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345343/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=040903a84b8e43cbaaed72bee8dd65c3 Command failed on smithi149 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345344] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:28:00 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345344/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec22f2eb7fb048648bc4e52fa81adaf9 Command failed on smithi143 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345345] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:24:58 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345345/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345345/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82ad7e095e214173b036900020702893 Command failed on smithi179 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345346] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:39:33 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345346/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6df9a15bac14bd0be7f2904fc94919c Command failed on smithi003 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345347] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:23:20 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345347/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345347/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2495a0bd46cb46918a37b9001c969783 Command failed on smithi114 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345348] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:29:17 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345348/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a93e408426f14c6ea30bdce3c9ee13d1 Command failed on smithi031 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345349] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:27:18 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345349/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=911907eec42a446baba9802c81f97337 Command failed on smithi023 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345350] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:29:56 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345350/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345350/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1de6a3cb572b4c47a4ea627d608569ac Command failed on smithi062 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345351] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:23:24 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345351/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c43200fc5df4e22b364817c2608245c Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345352] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:33:53 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345352/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d66ca428c81749b68d8289adadf0179c Command failed on smithi016 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345353] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:27:46 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345353/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb6ecc7d08f64e3997c7e6cd9adb7796 Command failed on smithi033 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345354] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:29:41 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345354/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8031687a4ff413a9c7c3497e1ea10cb Command failed on smithi078 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345355] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:35:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345355/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=080a5f40902d415d8fc9f41400943d1e Command failed on smithi018 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345356] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:31:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345356/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43c0d75d43ae473b81d21bd59ab58680 Command failed on smithi064 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345357] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:26:26 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345357/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4db07111ec78420c958ce76fc802daee Command failed on smithi037 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345358] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:30:44 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345358/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2646d0290154a63bfbdfa8ee1d7e0a5 Command failed on smithi087 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345359] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:24:02 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345359/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd4aa5f0a18444a8a1832c02544d1c5c Command failed on smithi070 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345360] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:29:26 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345360/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345360/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1eda642e2b374896a1f32f50a72ecd56 Command failed on smithi008 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345361] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:31:24 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345361/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345361/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ed8d60d5bbd471b8bbadddbee596970 Command failed on smithi026 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345362] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:34:02 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345362/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345362/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09e8470f5c62466ba9b807006229ab04 Command failed on smithi045 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345363] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:26:33 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345363/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345363/ sentry: https://sentry.ceph.com/organizations/ceph/?query=153814ab1ece4a04ac5c71775a6115fb Command failed on smithi069 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345364] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:29:45 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345364/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345364/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddd8c8e299cc413689797e8a0bd3a7b5 Command failed on smithi138 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345365] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:30:08 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345365/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95421ee5e0b74bddb5c3608fdee99aaa Command failed on smithi072 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345366] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:39:00 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345366/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345366/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d522c7cbd2944ebcafd39d5978af4c64 Command failed on smithi027 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345367] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:25:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345367/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f973bb44628545a691726a6d203e510c Command failed on smithi131 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345368] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:29:32 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345368/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345368/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4329c0cc0054a719c25a0aa5ce908ad Command failed on smithi006 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345369] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345369/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345369/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1f0728ec3924cfc9e1d7c2ba2620bcf Command failed on smithi050 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345370] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:33:48 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345370/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345370/ sentry: https://sentry.ceph.com/organizations/ceph/?query=863fbca2d6c74a6fa86123c3c7a8bb09 Command failed on smithi040 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345371] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:29:40 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345371/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62eeac9bbea44f8d89951ad899f8f779 Command failed on smithi022 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345372] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:29:15 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345372/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22a4e31ca16d447bb4139192bbcfaade Command failed on smithi179 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345373] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:24:53 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345373/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b179bbc0568947049e0787d890958996 Command failed on smithi098 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345374] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:29:14 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345374/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7823c50caa04455ab1c0dcacd74f83c Command failed on smithi079 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345375] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:31:27 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345375/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345375/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0eeeab93ca4541a1a24969cd11ee2d56 Command failed on smithi149 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345376] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:29:25 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345376/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345376/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59b39c984cdd486a992dfaf7fa998c91 Command failed on smithi002 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345377] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:28:55 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345377/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1982b879aefe4d1cb7f538cac871c8b1 Command failed on smithi114 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345378] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:36:01 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345378/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cff403579a4f44b2a0f4f23dcaa2d3db Command failed on smithi049 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345379] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:24:30 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345379/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7de9298e10894e938cf691ae6d467e45 Command failed on smithi044 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345380] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:31:10 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345380/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d90d965cdf94d7d9de120714abb9536 Command failed on smithi052 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345381] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:24:24 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345381/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70830be8ce0243b89cbf1723c9ee2b44 Command failed on smithi143 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345382] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:39:01 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345382/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3406c4245c543d887cce79216540562 Command failed on smithi112 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345383] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:25:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345383/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345383/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd0dd46a3d8446e194684c4c20812210 Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345384] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:30:48 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345384/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345384/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e555cdfe15741e884230d9de31b1836 Command failed on smithi023 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345385] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:24:48 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345385/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfe15ceb83624cc68110245825103f17 Command failed on smithi046 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345386] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:31:00 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345386/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345386/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d137837a27548cb84995349562921c4 Command failed on smithi007 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345387] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:31:32 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345387/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345387/ sentry: https://sentry.ceph.com/organizations/ceph/?query=637db0c62e4644b0894728e0e8143e08 Command failed on smithi019 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345388] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:30:13 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345388/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1d846ce17e9493d8bc77408f2caa7cb Command failed on smithi137 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345389] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:29:34 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345389/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345389/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e28974131b2435491fbbcade4517cb2 Command failed on smithi005 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345390] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:33:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345390/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345390/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5a5a5307a77640c2b09cf390afd230b1 Command failed on smithi037 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345391] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:29:40 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345391/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345391/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3542930bc1e241a189f585cbb7e82f57 Command failed on smithi033 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345392] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:34:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345392/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ac9497282ed4568b09b6f5b805a7ca3 Command failed on smithi070 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345393] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:25:38 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345393/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345393/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27f85ef9654b41b78f57965ec4107765 Command failed on smithi031 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345394] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:34:39 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345394/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=01965d1af2c94ed6b012d9386197645a Command failed on smithi003 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345395] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:29:05 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345395/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8f76bbb72994c40b1a913449c3e83a1 Command failed on smithi062 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345396] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:33:46 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345396/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345396/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd8aff194438440f86cea6680d9b5ae0 Command failed on smithi078 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345397] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:29:59 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345397/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a802c3576b2b46dda834eb8caad3035d Command failed on smithi069 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345398] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:30:48 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345398/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e798ac833e6422882cd2e8b6f1400a6 Command failed on smithi018 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345399] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:27:55 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345399/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345399/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa25321de40647d1b887955e1d731d7b Command failed on smithi016 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345400] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:30:22 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345400/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc77b3c1c5f447c59c9e1d9205273976 Command failed on smithi057 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345401] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:25:17 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345401/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9ffca6a26b54cc0945a945d7cbd51b1 Command failed on smithi064 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345402] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:30:19 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345402/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345402/ sentry: https://sentry.ceph.com/organizations/ceph/?query=496c3b1083d34bbab971cbfaddf58722 Command failed on smithi032 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345403] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:25:14 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345403/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345403/ sentry: https://sentry.ceph.com/organizations/ceph/?query=306ee00df5954d2e8dde34cd19afa285 Command failed on smithi131 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345404] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:38:15 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345404/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a400f856c1a04ea4894f22731f19aaa1 Command failed on smithi026 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345405] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:25:04 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345405/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345405/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e75428c75da4f4a9c961e366f8ec2b7 Command failed on smithi053 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345406] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:56 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345406/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c77fb22342f64ba9b8b32020667549e4 Command failed on smithi072 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345407] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345407/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbdf59db44ce4d8dac4c2dbe50c75891 Command failed on smithi098 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345408] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:32:59 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345408/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345408/ sentry: https://sentry.ceph.com/organizations/ceph/?query=449e19e9f08047fa83f65a8ac525b2b9 Command failed on smithi008 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345409] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:32:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345409/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345409/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ccc8d11710e464c84de63c392147809 Command failed on smithi050 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345410] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:34:53 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345410/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1e3431a8ff834ba097259af344f6e415 Command failed on smithi044 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345411] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:30:04 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345411/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345411/ sentry: https://sentry.ceph.com/organizations/ceph/?query=593a305e5a634197baa7a3aa81e45b5a Command failed on smithi006 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345412] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:38:38 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345412/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345412/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87ac63f6137c4fd28e857613740e4a36 Command failed on smithi138 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345413] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:26:06 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345413/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20048ca306be40808abc656bf03f4db2 Command failed on smithi022 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345414] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:31:35 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345414/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345414/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecbd00c8753f4ec5a03f2a296f06e61e Command failed on smithi045 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345415] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:32:45 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345415/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345415/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d0830c6625349ea8045c2f34f84df2a Command failed on smithi143 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345417] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:32:35 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5aea0705bbf84ad9b39b50b522198299 Command failed on smithi061 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345418] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:29:21 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345418/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76d21396d0d5414a999b24feb276db59 Command failed on smithi114 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345419] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:27:49 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345419/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345419/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da00303bfe1548beb8e7a666f1783b65 Command failed on smithi017 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345420] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:30:07 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345420/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a520b87113a44b7b3b51892b6f30e2c Command failed on smithi162 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345421] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:30:09 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345421/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345421/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c2ceeadc80148dca1a453441234b8cd Command failed on smithi079 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345422] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:36:44 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345422/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f431dced3d14cc3ad5fa0e78f91a016 Command failed on smithi046 with status 2: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0' [7345423] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:19:53 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345423/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345423/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e8485c89a63b40fd91413610b6eeb55b Command failed on smithi135 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:1b0b2eb005a7c51d97a788b9900d511e1394b537 pull' Dead ================================================================= [7345416] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345416/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-07-20_16:47:59-fs:workload-wip-pdonnell-testing-20230719.201220-distro-default-smithi/7345416/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds