Found 6 jobs Missing teuthology log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317704/teuthology.log Missing teuthology log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317704/teuthology.log Ignoring malparsed backtrace: ceph version 18.0.0-4621-g799c6bac (799c6bac9b54face2c9a8fa291d4dea554b4f858) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5622e9eb43c0] , 2: (KernelDevice::_aio_thread()+0x9bc) [0x5622ea76ebec] Ignoring malparsed backtrace: 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f36c0069430 0x7f36c01f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36c00cc220 0x7f36c00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36c015baa0 0x7f36c00b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect Ignoring malparsed backtrace: 2023-06-27T10:19:23.350+0000 7efd42ffd640 1 -- 172.21.15.60:0/513481191 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6440+0+0 (secure 0 0 0) 0x7efd3c0831f0 con 0x7efd4c1560d0 , 2023-06-27T10:19:23.354+0000 7efd52038640 1 --2- 172.21.15.60:0/513481191 >> [v2:172.21.15.97:6832/2495490075,v1:172.21.15.97:6833/2495490075] conn(0x7efd30066350 0x7efd30068800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-06-27T10:19:23.354+0000 7efd52038640 1 --2- 172.21.15.60:0/513481191 >> [v2:172.21.15.97:6832/2495490075,v1:172.21.15.97:6833/2495490075] conn(0x7efd30066350 0x7efd30068800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efd4c15dfe0 tx=0x7efd48008040 comp rx=0 tx=0).ready entity=mgr.4123 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2023-06-27T10:19:25.866+0000 7f3542d00640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0ce990 con 0x7f353c15bc00 , 2023-06-27T10:19:25.866+0000 7f353bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f353c0b1150 0x7f353c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-06-27T10:19:25.866+0000 7f353bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f353c0b1150 0x7f353c0cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40778/0 (socket says 172.21.15.60:40778) Ignoring malparsed backtrace: 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1900d02a0 0x7fb1901548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb190155070 0x7fb190156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014f0c0 con 0x7fb1900b1150 Ignoring malparsed backtrace: 2023-06-27T10:19:27.346+0000 7f7b36ffd640 1 -- 172.21.15.60:0/1754397302 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169708 (secure 0 0 0) 0x7f7b4015cd60 con 0x7f7b4006a540 , 2023-06-27T10:19:27.646+0000 7f7b485fa640 1 -- 172.21.15.60:0/1754397302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7b4004d790 con 0x7f7b4006a540 , 2023-06-27T10:19:27.646+0000 7f7b36ffd640 1 -- 172.21.15.60:0/1754397302 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7b4015cd60 con 0x7f7b4006a540 Ignoring malparsed backtrace: nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 , , 2023-06-27T10:19:28.241 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com Ignoring malparsed backtrace: ceph-osd: no process found , 2023-06-27T10:29:16.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:29:16.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T10:54:23.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:54:23.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:54:53.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:19:28.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:19:58.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:19:58.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:45:04.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:45:04.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:45:35.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:10:10.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:10:41.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:10:41.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:35:46.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:35:46.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:36:16.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:00:51.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:01:21.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:01:21.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:26:26.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:26:26.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:26:56.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:51:31.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:52:01.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:52:01.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:17:07.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:17:07.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:17:37.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:42:12.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:42:42.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:42:42.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:07:47.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:07:47.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:08:17.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:32:53.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:33:23.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:33:23.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:58:26.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:58:26.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:58:56.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:23:30.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:24:00.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:24:00.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:49:04.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:49:04.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:49:34.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:14:08.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:14:38.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:14:38.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:39:42.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:39:42.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:40:12.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:04:46.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:05:16.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:05:16.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:30:23.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:30:23.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:30:53.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317706/remote/smithi060/log/ceph-osd.0.log.gz Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317706/remote/smithi060/log/ceph-mgr.y.log.gz Not-understood crash indication Ignoring malparsed backtrace: ceph version 18.0.0-4621-g799c6bac (799c6bac9b54face2c9a8fa291d4dea554b4f858) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5622e9eb43c0] , 2: (KernelDevice::_aio_thread()+0x9bc) [0x5622ea76ebec] Ignoring malparsed backtrace: 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f36c0069430 0x7f36c01f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f36c00cc220 0x7f36c00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:20.507+0000 7f36c7f03640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36c015baa0 0x7f36c00b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect Ignoring malparsed backtrace: 2023-06-27T10:19:23.350+0000 7efd42ffd640 1 -- 172.21.15.60:0/513481191 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6440+0+0 (secure 0 0 0) 0x7efd3c0831f0 con 0x7efd4c1560d0 , 2023-06-27T10:19:23.354+0000 7efd52038640 1 --2- 172.21.15.60:0/513481191 >> [v2:172.21.15.97:6832/2495490075,v1:172.21.15.97:6833/2495490075] conn(0x7efd30066350 0x7efd30068800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-06-27T10:19:23.354+0000 7efd52038640 1 --2- 172.21.15.60:0/513481191 >> [v2:172.21.15.97:6832/2495490075,v1:172.21.15.97:6833/2495490075] conn(0x7efd30066350 0x7efd30068800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efd4c15dfe0 tx=0x7efd48008040 comp rx=0 tx=0).ready entity=mgr.4123 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 Ignoring malparsed backtrace: 2023-06-27T10:19:25.866+0000 7f3542d00640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c0ce990 con 0x7f353c15bc00 , 2023-06-27T10:19:25.866+0000 7f353bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f353c0b1150 0x7f353c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 , 2023-06-27T10:19:25.866+0000 7f353bfff640 1 --2- >> v2:172.21.15.60:3300/0 conn(0x7f353c0b1150 0x7f353c0cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40778/0 (socket says 172.21.15.60:40778) Ignoring malparsed backtrace: 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb1900d02a0 0x7fb1901548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb190155070 0x7fb190156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect , 2023-06-27T10:19:26.778+0000 7fb1952ca640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014f0c0 con 0x7fb1900b1150 Ignoring malparsed backtrace: 2023-06-27T10:19:27.346+0000 7f7b36ffd640 1 -- 172.21.15.60:0/1754397302 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169708 (secure 0 0 0) 0x7f7b4015cd60 con 0x7f7b4006a540 , 2023-06-27T10:19:27.646+0000 7f7b485fa640 1 -- 172.21.15.60:0/1754397302 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7b4004d790 con 0x7f7b4006a540 , 2023-06-27T10:19:27.646+0000 7f7b36ffd640 1 -- 172.21.15.60:0/1754397302 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7b4015cd60 con 0x7f7b4006a540 Ignoring malparsed backtrace: nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 , , 2023-06-27T10:19:28.241 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi097.front.sepia.ceph.com Ignoring malparsed backtrace: ceph-osd: no process found , 2023-06-27T10:29:16.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:29:16.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T10:54:23.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:54:23.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T10:54:53.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:19:28.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:19:58.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:19:58.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T11:45:04.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:45:04.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T11:45:35.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:10:10.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:10:41.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:10:41.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T12:35:46.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:35:46.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T12:36:16.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:00:51.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:01:21.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:01:21.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:26:26.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:26:26.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:26:56.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T13:51:31.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:52:01.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T13:52:01.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:17:07.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:17:07.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:17:37.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T14:42:12.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:42:42.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T14:42:42.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:07:47.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:07:47.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:08:17.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:32:53.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:33:23.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:33:23.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T15:58:26.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:58:26.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T15:58:56.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:23:30.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:24:00.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:24:00.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T16:49:04.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:49:04.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T16:49:34.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:14:08.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:14:38.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:14:38.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T17:39:42.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:39:42.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T17:40:12.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:04:46.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:05:16.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:05:16.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf Ignoring malparsed backtrace: 2023-06-27T18:30:23.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:30:23.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf , 2023-06-27T18:30:53.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo logrotate /etc/logrotate.d/ceph-test.conf Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317706/remote/smithi060/log/ceph-osd.0.log.gz Missing log /home/teuthworker/archive/xiubli-2023-06-27_07:14:25-fs:functional-wip-lxb-xlock-20230627-1253-distro-default-smithi/7317706/remote/smithi060/log/ceph-mgr.y.log.gz Not-understood crash indication Found 5 distinct failure reasons Dead: b'2023-06-27T19:15:55.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf' ceph version 18.0.0-4621-g799c6bac (799c6bac9b54face2c9a8fa291d4dea554b4f858) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f949267ece0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f94938cbe19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f94938cbf85] 6: ceph-mds(+0x3ce6c2) [0x55dd864b76c2] 7: (Locker::xlock_finish(std::_Rb_tree_const_iterator const&, MutationImpl*, bool*)+0x1ca) [0x55dd864b962a] 8: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0x3f7) [0x55dd864c20b7] 9: (Locker::drop_non_rdlocks(MutationImpl*, std::set, std::allocator >*)+0x5f) [0x55dd864c693f] 10: (Server::reply_client_request(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x497) [0x55dd86318d57] 11: (Server::respond_to_request(boost::intrusive_ptr&, int)+0x350) [0x55dd86319c00] 12: (Server::_unlink_local_finish(boost::intrusive_ptr&, CDentry*, CDentry*, unsigned long)+0x2ea) [0x55dd86337dba] 13: (MDSContext::complete(int)+0x5f) [0x55dd8660001f] 14: (MDSIOContextBase::complete(int)+0x524) [0x55dd866007a4] 15: (MDSLogContextBase::complete(int)+0x5b) [0x55dd86600b5b] 16: (Finisher::finisher_thread_entry()+0x18d) [0x7f949396f85d] 17: /lib64/libpthread.so.0(+0x81cf) [0x7f94926741cf] 18: clone() 1 jobs: ['7317705'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] Dead: None 1 jobs: ['7317704'] Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2 jobs: ['7317702', '7317701'] suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}'] Dead: b'2023-06-27T19:16:01.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 1 jobs: ['7317706'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] Dead: b'2023-06-27T14:17:35.703 INFO:tasks.ceph.osd.6.smithi192.stderr:problem writing to /var/log/ceph/ceph-osd.6.log: (28) No space left on device' 1 jobs: ['7317703'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']