2023-03-29T10:32:41.836 INFO:root:teuthology version: 0.0.1.dev78+ge372916 2023-03-29T10:32:41.836 INFO:teuthology.results:subset: '816/1024' 2023-03-29T10:32:41.836 INFO:teuthology.results:seed: '9480' 2023-03-29T10:33:45.059 INFO:root:teuthology version: 0.0.1.dev78+ge372916 2023-03-29T10:33:45.062 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-29T22:34:41.429 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-29T22:34:41.702 INFO:scrape:Found 3 jobs 2023-03-29T22:34:42.733 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-3066-gf03d69a8 (f03d69a8741c605ae6711afe2c9991ac46095f7d) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x560ff206accb] , 2: (KernelDevice::_aio_thread()+0x1798) [0x560ff2c7a908] 2023-03-29T22:34:42.734 WARNING:scrape:Ignoring malparsed backtrace: nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 , nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 , nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-03-29T22:34:42.734 WARNING:scrape:Ignoring malparsed backtrace: tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 , nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 , nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 2023-03-29T22:34:42.734 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T15:33:25.324+0000 7f79c5fff700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 110133) UID: 0 , 2023-03-29T15:33:25.324+0000 7f79c5fff700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** , 2023-03-29T15:33:25.326+0000 7f155ddf9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 128392) UID: 0 2023-03-29T22:34:42.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T15:55:58.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T15:56:29.095 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T15:56:29.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T16:21:40.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T16:21:40.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T16:22:10.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T16:46:52.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T16:47:22.673 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T16:47:22.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T17:12:34.631 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T17:12:34.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T17:13:04.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T17:37:45.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T17:38:16.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T17:38:16.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T18:03:28.106 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:03:28.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:03:58.312 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T18:28:39.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:29:10.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:29:10.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T18:54:22.203 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:54:22.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T18:54:52.476 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.736 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T19:19:34.513 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T19:20:04.780 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T19:20:04.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T19:45:16.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T19:45:16.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T19:45:46.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T20:10:28.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T20:10:58.384 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T20:10:58.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T20:36:10.037 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T20:36:10.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T20:36:40.216 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T21:01:21.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:01:51.546 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:01:51.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T21:27:02.604 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:27:02.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:27:32.818 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-29T21:52:13.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:52:44.023 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-03-29T21:52:44.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-29T22:34:42.755 WARNING:scrape:Missing log /home/teuthworker/archive/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225143/remote/smithi111/log/ceph-osd.1.log.gz 2023-03-29T22:34:42.755 WARNING:scrape:Missing log /home/teuthworker/archive/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225143/remote/smithi119/log/ceph-mgr.x.log.gz 2023-03-29T22:34:42.755 WARNING:scrape:Not-understood crash indication 2023-03-29T22:34:42.755 INFO:scrape:Found 2 distinct failure reasons 2023-03-29T22:34:42.756 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2023-03-29T22:34:42.756 INFO:scrape:1 jobs: ['7225144'] 2023-03-29T22:34:42.756 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-03-29T22:34:42.756 INFO:scrape: 2023-03-29T22:34:42.756 INFO:scrape:Dead: b'2023-03-29T22:34:32.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-03-29T22:34:42.756 INFO:scrape:1 jobs: ['7225143'] 2023-03-29T22:34:42.756 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-03-29T22:34:42.756 INFO:scrape: 2023-03-29T22:34:42.757 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/ failed: 1 dead: 0 running: 1 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7225144] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:26:42 info: http://pulpito.ceph.com/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225144/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225144/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f67626043a1b48a8a7771250a32aff27 Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) Running ================================================================= [7225143] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} info: http://pulpito.ceph.com/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225143/ Pass ================================================================= [7225145] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:43:44 info: http://pulpito.ceph.com/vshankar-2023-03-29_10:31:35-fs-wip-vshankar-testing-20230317.095222-testing-default-smithi/7225145/