2023-04-05T19:29:11.682 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-05T19:29:11.683 INFO:teuthology.results:subset: '24/128' 2023-04-05T19:29:11.683 INFO:teuthology.results:seed: '7623' 2023-04-05T19:32:45.846 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-05T19:32:45.850 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-05T20:49:51.804 INFO:teuthology.results:Tests finished! gathering results... 2023-04-05T20:49:52.113 INFO:scrape:Found 4 jobs 2023-04-05T20:49:54.557 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:54.738 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:54.919 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:55.100 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:55.280 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:55.459 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f5583381d55] , 2: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 2023-04-05T20:49:55.460 INFO:scrape:Found 4 distinct failure reasons 2023-04-05T20:49:55.460 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-04-05T20:49:55.460 INFO:scrape:1 jobs: ['7233031'] 2023-04-05T20:49:55.461 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2023-04-05T20:49:55.461 INFO:scrape: 2023-04-05T20:49:55.461 INFO:scrape:Failure: "2023-04-05T20:08:21.169538+0000 mon.a (mon.0) 516 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-04-05T20:49:55.462 INFO:scrape:1 jobs: ['7233029'] 2023-04-05T20:49:55.462 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2023-04-05T20:49:55.462 INFO:scrape: 2023-04-05T20:49:55.462 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2023-04-05T20:49:55.463 INFO:scrape:1 jobs: ['7233030'] 2023-04-05T20:49:55.463 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/alternate-pool}'] 2023-04-05T20:49:55.463 INFO:scrape: 2023-04-05T20:49:55.464 INFO:scrape:Crash: Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=44b6581f04c8d41f47aa0216a156fa52bb6cba23 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh' 2023-04-05T20:49:55.464 INFO:scrape:ceph version 16.2.11-383-gcaeaa2e3 (caeaa2e33ca3e818edceb3fc8d7950e9ebf75f1e) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5582e5f420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f5583381db0] 5: /usr/lib/ceph/libceph-common.so.2(+0x267f5d) [0x7f5583381f5d] 6: (Monitor::~Monitor()+0x146b) [0x5632a7832d1b] 7: (Monitor::~Monitor()+0xd) [0x5632a7832d9d] 8: main() 9: __libc_start_main() 10: _start() 2023-04-05T20:49:55.464 INFO:scrape:1 jobs: ['7233028'] 2023-04-05T20:49:55.464 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/fs/misc}', 'wsync/{no}}'] 2023-04-05T20:49:55.465 INFO:scrape: 2023-04-05T20:49:55.465 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7233028] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 01:09:07 info: http://pulpito.ceph.com/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233028/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4e1f6b08dec4e32bd1c74f0b0d3a9ce Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi103 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=44b6581f04c8d41f47aa0216a156fa52bb6cba23 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/misc/multiple_rsync.sh' [7233029] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:48:22 info: http://pulpito.ceph.com/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233029/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233029/ "2023-04-05T20:08:21.169538+0000 mon.a (mon.0) 516 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7233030] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:11:42 info: http://pulpito.ceph.com/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233030/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233030/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08bfd877ac5b4277b99b10155b15a984 Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7233031] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:24:10 info: http://pulpito.ceph.com/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233031/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-05_19:27:46-fs-wip-pdonnell-testing-20230330.155904-pacific-distro-default-smithi/7233031/ sentry: https://sentry.ceph.com/organizations/ceph/?query=074569d8925d4620a6be2283962231a1 Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery)