Found 6 jobs Ignoring malparsed backtrace: ceph version 18.0.0-4621-g799c6bac (799c6bac9b54face2c9a8fa291d4dea554b4f858) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55bf28833cf5] , 2: (KernelDevice::_aio_thread()+0x1798) [0x55bf29446fa8] Ignoring malparsed backtrace: 2023-06-27T10:07:51.744+0000 7f7e65855700 1 --2- 172.21.15.98:0/55883365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e6006d760 0x7f7e60070eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-06-27T10:07:51.744+0000 7f7e65855700 1 -- 172.21.15.98:0/55883365 >> v2:172.21.15.145:3300/0 conn(0x7f7e6015d200 msgr2=0x7f7e6006e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-06-27T10:07:51.744+0000 7f7e65855700 1 --2- 172.21.15.98:0/55883365 >> v2:172.21.15.145:3300/0 conn(0x7f7e6015d200 0x7f7e6006e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop Ignoring malparsed backtrace: 2023-06-27T10:07:52.322+0000 7f3e664f0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e600cbe30 con 0x7f3e6016c970 , 2023-06-27T10:07:52.322+0000 7f3e664f0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60168ee0 con 0x7f3e600cb4f0 , 2023-06-27T10:07:52.322+0000 7f3e64a8d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e6016c970 0x7f3e600cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 Ignoring malparsed backtrace: 2023-06-27T10:07:53.216+0000 7facaf1f6700 1 -- 172.21.15.98:0/344872995 >> [v2:172.21.15.98:6832/2785724915,v1:172.21.15.98:6833/2785724915] conn(0x7fac8c06ace0 msgr2=0x7fac8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down , 2023-06-27T10:07:53.216+0000 7facaf1f6700 1 --2- 172.21.15.98:0/344872995 >> [v2:172.21.15.98:6832/2785724915,v1:172.21.15.98:6833/2785724915] conn(0x7fac8c06ace0 0x7fac8c06d190 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fac9c005910 tx=0x7fac9c009040 comp rx=0 tx=0).stop , 2023-06-27T10:07:53.217+0000 7facaf1f6700 1 -- 172.21.15.98:0/344872995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faca815d280 msgr2=0x7faca81f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down Ignoring malparsed backtrace: 2023-06-27T10:07:54.000+0000 7f2f4f7fe700 1 --2- 172.21.15.98:0/1038609143 >> v2:172.21.15.145:3300/0 conn(0x7f2f5006c950 0x7f2f500c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop , 2023-06-27T10:07:54.000+0000 7f2f4f7fe700 1 -- 172.21.15.98:0/1038609143 >> v1:172.21.15.145:6789/0 conn(0x7f2f5006d990 legacy=0x7f2f5015ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down , 2023-06-27T10:07:54.000+0000 7f2f4f7fe700 1 -- 172.21.15.98:0/1038609143 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5006a7c0 con 0x7f2f5006cdc0 Ignoring malparsed backtrace: 2023-06-27T10:07:54.589+0000 7f86be35c700 1 Processor -- start , 2023-06-27T10:07:54.589+0000 7f86be35c700 1 -- start start , 2023-06-27T10:07:54.590+0000 7f86be35c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86b8069ef0 0x7f86b81522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect Ignoring malparsed backtrace: 2023-06-27T10:07:55.440 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) , 2023-06-27T10:07:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 , rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory Ignoring malparsed backtrace: , 2023-06-27T10:07:55.693 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com , 2023-06-27T10:07:55.693 INFO:teuthology.orchestra.run:Running command with timeout 300 Ignoring malparsed backtrace: 2023-06-27T10:20:15.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:20:15.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:20:46.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T10:45:23.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:45:53.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:45:53.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:11:00.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:11:00.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:11:30.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:36:08.554 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:36:38.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:36:38.681 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:01:46.176 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:01:46.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:02:16.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:26:53.649 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:27:23.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:27:23.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:52:31.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:52:31.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:53:01.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:17:38.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:18:08.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:18:08.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:43:16.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:43:16.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:43:46.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:08:23.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:08:53.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:08:53.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:34:01.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:34:01.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:34:31.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:59:09.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:59:39.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:59:39.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:24:46.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:24:46.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:25:16.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:49:54.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:50:24.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:50:24.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:15:32.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:15:32.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:16:02.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:40:39.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:41:09.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:41:09.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:06:17.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:06:17.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:06:47.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:31:23.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:31:53.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:31:53.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:57:01.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:57:01.330 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:57:31.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:22:08.437 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:22:38.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:22:38.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:47:45.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:47:45.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:48:15.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:13:53-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317688/remote/smithi098/log/ceph-osd.0.log.gz Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:13:53-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317688/remote/smithi145/log/ceph-mgr.x.log.gz Not-understood crash indication Found 3 distinct failure reasons Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 3 jobs: ['7317686', '7317689', '7317685'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}'] Dead: b'2023-06-27T19:14:52.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2 jobs: ['7317688', '7317687'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}'] Dead: b'2023-06-27T19:13:02.659 INFO:tasks.ceph.osd.7.smithi149.stderr:2023-06-27T19:13:02.655+0000 7f3f297f9700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 160465) UID: 0' 1 jobs: ['7317690'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']