2022-07-02T16:45:43.427 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-02T16:45:43.428 INFO:teuthology.results:subset: '1/999' 2022-07-02T16:45:43.428 INFO:teuthology.results:seed: '6408' 2022-07-02T17:33:02.780 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-02T17:33:02.805 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-03T05:33:03.431 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-07-03T05:33:07.504 INFO:scrape:Found 48 jobs 2022-07-03T05:33:11.700 INFO:scrape:Found 7 distinct failure reasons 2022-07-03T05:33:11.701 INFO:scrape:Failure: "2022-07-02T18:05:58.438533+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-07-03T05:33:11.702 INFO:scrape:1 jobs: ['6909654'] 2022-07-03T05:33:11.702 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snap-schedule}'] 2022-07-03T05:33:11.703 INFO:scrape: 2022-07-03T05:33:11.703 INFO:scrape:Dead: b'2022-07-03T05:07:03.969 DEBUG:teuthology.exit:Finished running handlers' 2022-07-03T05:33:11.703 INFO:scrape:2 jobs: ['6909637', '6909651'] 2022-07-03T05:33:11.704 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2022-07-03T05:33:11.704 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/scrub}', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-07-03T05:33:11.705 INFO:scrape: 2022-07-03T05:33:11.705 INFO:scrape:Dead: b'2022-07-03T05:33:08.306 INFO:tasks.ceph.osd.1.smithi139.stderr:2022-07-03T05:33:08.303+0000 7fdd902d8700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-07-03T04:33:08.307868+0000)' 2022-07-03T05:33:11.706 INFO:scrape:1 jobs: ['6909674'] 2022-07-03T05:33:11.706 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-07-03T05:33:11.707 INFO:scrape: 2022-07-03T05:33:11.707 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-07-03T05:33:11.708 INFO:scrape:1 jobs: ['6909666'] 2022-07-03T05:33:11.708 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-07-03T05:33:11.708 INFO:scrape: 2022-07-03T05:33:11.709 INFO:scrape:Failure: Command failed on smithi026 with status 123: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2022-07-03T05:33:11.709 INFO:scrape:1 jobs: ['6909653'] 2022-07-03T05:33:11.710 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2022-07-03T05:33:11.710 INFO:scrape: 2022-07-03T05:33:11.711 INFO:scrape:Dead: b'2022-07-02T18:02:08.801 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2022-07-03T05:33:11.711 INFO:scrape:1 jobs: ['6909668'] 2022-07-03T05:33:11.712 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}'] 2022-07-03T05:33:11.712 INFO:scrape: 2022-07-03T05:33:11.713 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2022-07-03T05:33:11.713 INFO:scrape:1 jobs: ['6909669'] 2022-07-03T05:33:11.713 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2022-07-03T05:33:11.714 INFO:scrape: 2022-07-03T05:33:11.716 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/ failed: 4 dead: 2 running: 2 waiting: 0 queued: 0 passed: 40 Fail ================================================================= [6909653] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:01:11 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909653/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909653/ Command failed on smithi026 with status 123: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [6909654] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:46:52 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909654/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909654/ "2022-07-02T18:05:58.438533+0000 mon.a (mon.0) 1862 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6909666] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:19:03 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909666/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909666/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f77aa08b8e79440cb782da10e207df91 reached maximum tries (90) after waiting for 540 seconds [6909669] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:12:25 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909669/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909669/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2badb06a57b240c8b75553458971e7d3 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) Dead ================================================================= [6909637] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909637/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909637/ hit max job timeout [6909651] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909651/ log: http://qa-proxy.ceph.com/teuthology/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909651/ hit max job timeout Running ================================================================= [6909668] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909668/ [6909674] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909674/ Pass ================================================================= [6909627] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:48:38 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909627/ [6909628] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:08:15 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909628/ [6909629] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:16:06 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909629/ [6909630] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:25 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909630/ [6909631] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:36:32 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909631/ [6909632] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:23:16 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909632/ [6909633] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:08 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909633/ [6909634] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:26:37 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909634/ [6909635] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:13:36 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909635/ [6909636] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:43:43 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909636/ [6909638] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:13:10 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909638/ [6909639] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:05:51 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909639/ [6909640] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:15:21 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909640/ [6909641] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:12:33 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909641/ [6909642] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:10 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909642/ [6909643] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:36 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909643/ [6909644] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:17:36 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909644/ [6909645] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:14 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909645/ [6909646] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:57:47 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909646/ [6909647] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:39 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909647/ [6909648] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:15:34 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909648/ [6909649] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:11:39 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909649/ [6909650] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:12:21 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909650/ [6909652] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:14:27 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909652/ [6909655] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:49:42 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909655/ [6909656] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:45 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909656/ [6909657] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:13:09 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909657/ [6909658] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:57 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909658/ [6909659] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:00 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909659/ [6909660] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:15:22 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909660/ [6909661] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:19 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909661/ [6909662] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:43 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909662/ [6909663] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:17:30 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909663/ [6909664] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:11:31 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909664/ [6909665] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:12:14 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909665/ [6909667] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:22:47 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909667/ [6909670] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:13:54 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909670/ [6909671] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:42:49 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909671/ [6909672] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:02 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909672/ [6909673] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:50:10 info: http://pulpito.ceph.com/adking-2022-07-01_20:15:02-fs-wip-adk-testing-2022-06-30-2110-distro-default-smithi/6909673/