2023-08-01T07:37:42.725 INFO:root:teuthology version: 0.0.1.dev174+g10753fd 2023-08-01T07:37:42.726 INFO:teuthology.results:subset: '111/120000' 2023-08-01T07:37:42.726 INFO:teuthology.results:seed: '7177' 2023-08-01T07:38:38.513 INFO:root:teuthology version: 0.0.1.dev174+g10753fd 2023-08-01T07:38:38.518 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-01T10:19:49.311 INFO:teuthology.results:Tests finished! gathering results... 2023-08-01T10:19:50.175 INFO:scrape:Found 11 jobs 2023-08-01T10:19:53.171 WARNING:scrape:Misunderstood line: 2023-08-01T10:19:57.318 INFO:scrape:Found 7 distinct failure reasons 2023-08-01T10:19:57.319 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-08-01T10:19:57.319 INFO:scrape:2 jobs: ['7357923', '7357919'] 2023-08-01T10:19:57.320 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-08-01T10:19:57.320 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{rhel_8}', 'tasks/{0-subvolume/{with-no-extra-options}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-08-01T10:19:57.320 INFO:scrape: 2023-08-01T10:19:57.321 INFO:scrape:Failure: Command failed on smithi079 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-08-01T10:19:57.321 INFO:scrape:4 jobs: ['7357922', '7357928', '7357929', '7357921'] 2023-08-01T10:19:57.321 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-08-01T10:19:57.321 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-08-01T10:19:57.322 INFO:scrape: 2023-08-01T10:19:57.322 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-08-01T10:19:57.322 INFO:scrape:1 jobs: ['7357920'] 2023-08-01T10:19:57.323 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-08-01T10:19:57.323 INFO:scrape: 2023-08-01T10:19:57.323 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-08-01T10:19:57.323 INFO:scrape:1 jobs: ['7357927'] 2023-08-01T10:19:57.324 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-08-01T10:19:57.324 INFO:scrape: 2023-08-01T10:19:57.324 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi090 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1d9818965953d05a13482bac86d113bd06a197b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-08-01T10:19:57.325 INFO:scrape:1 jobs: ['7357924'] 2023-08-01T10:19:57.325 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-08-01T10:19:57.325 INFO:scrape: 2023-08-01T10:19:57.326 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi204 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1d9818965953d05a13482bac86d113bd06a197b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' 2023-08-01T10:19:57.326 INFO:scrape:1 jobs: ['7357926'] 2023-08-01T10:19:57.326 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-iozone}'] 2023-08-01T10:19:57.327 INFO:scrape: 2023-08-01T10:19:57.327 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-08-01T10:19:57.327 INFO:scrape:1 jobs: ['7357925'] 2023-08-01T10:19:57.327 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-08-01T10:19:57.328 INFO:scrape: 2023-08-01T10:19:57.329 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/ failed: 11 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7357919] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:47:38 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357919/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357919/ reached maximum tries (51) after waiting for 300 seconds [7357920] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:46:31 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357920/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9ab2d96a97543a9b51f8894acc9e70f Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7357921] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:18:09 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357921/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fab60aac17d4350a2333469633db764 Command failed on smithi173 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7357922] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:35 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357922/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84d14a4d7e6442a2b4ca5745a20a7942 Command failed on smithi079 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7357923] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:37:53 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357923/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357923/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7357924] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:21:49 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357924/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ade6a49246b141359b65ca36ee5cf1a8 Command failed (workunit test libcephfs/test.sh) on smithi090 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1d9818965953d05a13482bac86d113bd06a197b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7357925] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:36:14 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357925/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357925/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85c4403d2a624154a0575e1c72aa00ed Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) [7357926] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 00:19:58 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357926/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c24bb7db4ced448cb30ea7c68ae14f75 Command failed (workunit test fs/fscrypt.sh) on smithi204 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a1d9818965953d05a13482bac86d113bd06a197b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' [7357927] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:08:46 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357927/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aed4705e6eb5472cb39636ae7e89a16c saw valgrind issues [7357928] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:17:39 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357928/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fd34242f8c640e2a5eea7c4f378d855 Command failed on smithi143 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7357929] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:18:04 info: http://pulpito.ceph.com/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357929/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-01_07:34:08-fs-wip-yuri-testing-2023-07-25-0833-reef-testing-default-smithi/7357929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fecba9bf1d09433d89c25f15d8066b97 Command failed on smithi077 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'