2023-07-26T20:14:57.290 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-26T20:14:57.291 INFO:teuthology.results:subset: '111/120000'
2023-07-26T20:14:57.291 INFO:teuthology.results:seed: '4711'
2023-07-26T21:47:03.676 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-26T21:47:03.711 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-27T04:33:01.642 INFO:teuthology.results:Tests finished! gathering results...
2023-07-27T04:33:04.327 INFO:scrape:Found 35 jobs
2023-07-27T04:33:04.698 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-26_14:33:03-rados-reef-release-distro-default-smithi/7353097/teuthology.log
2023-07-27T04:33:04.890 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.1.2-175-gc1444501 (c1444501ab7918ce42bdc26b9d860ad26e34dd69) reef (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x562336]
,  2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x25a) [0xb9961a]

2023-07-27T04:33:04.890 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:12:31.336 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.396 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.359 85287== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.891 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.Whiteout (10402 ms)
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8942 ms)

2023-07-27T04:33:04.891 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:bd63b0f1:::8:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e960b815:::9:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:52ea6a34:::10:head

2023-07-27T04:33:04.892 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e3de589f:::109:head
, 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:90f90a5b:::110:head
, 2023-07-26T22:41:24.123 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:a7b4f1d7:::111:head

2023-07-27T04:33:04.892 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:14:36.704 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.757 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.760 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.893 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:15:16.869 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.872 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.938 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.915 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.1.2-175-gc1444501 (c1444501ab7918ce42bdc26b9d860ad26e34dd69) reef (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x562336]
,  2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x25a) [0xb9961a]

2023-07-27T04:33:04.916 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:12:31.336 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.396 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.359 85287== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.916 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.Whiteout (10402 ms)
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8942 ms)

2023-07-27T04:33:04.917 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:bd63b0f1:::8:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e960b815:::9:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:52ea6a34:::10:head

2023-07-27T04:33:04.917 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e3de589f:::109:head
, 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:90f90a5b:::110:head
, 2023-07-26T22:41:24.123 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:a7b4f1d7:::111:head

2023-07-27T04:33:04.918 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:14:36.704 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.757 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.760 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.918 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:15:16.869 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.872 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.938 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.940 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.1.2-175-gc1444501 (c1444501ab7918ce42bdc26b9d860ad26e34dd69) reef (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x562336]
,  2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x25a) [0xb9961a]

2023-07-27T04:33:04.940 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:12:31.336 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.396 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.359 85287== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.Whiteout (10402 ms)
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8942 ms)

2023-07-27T04:33:04.941 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:bd63b0f1:::8:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e960b815:::9:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:52ea6a34:::10:head

2023-07-27T04:33:04.942 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e3de589f:::109:head
, 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:90f90a5b:::110:head
, 2023-07-26T22:41:24.123 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:a7b4f1d7:::111:head

2023-07-27T04:33:04.942 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:14:36.704 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.757 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.760 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.943 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:15:16.869 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.872 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.938 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.964 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.1.2-175-gc1444501 (c1444501ab7918ce42bdc26b9d860ad26e34dd69) reef (rc)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x562336]
,  2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x25a) [0xb9961a]

2023-07-27T04:33:04.965 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:12:31.336 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.396 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:12:31.359 85287== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.965 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.Whiteout (10402 ms)
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [ RUN      ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate
, 2023-07-26T22:41:02.510 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: [       OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8942 ms)

2023-07-27T04:33:04.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:bd63b0f1:::8:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e960b815:::9:head
, 2023-07-26T22:41:24.109 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:52ea6a34:::10:head

2023-07-27T04:33:04.966 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:e3de589f:::109:head
, 2023-07-26T22:41:24.122 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:90f90a5b:::110:head
, 2023-07-26T22:41:24.123 INFO:tasks.workunit.client.0.smithi066.stdout:              api_tier_pp: checking for 201:a7b4f1d7:::111:head

2023-07-27T04:33:04.967 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:14:36.704 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.757 85753== Warning: unimplemented fcntl command: 1036
, ==00:00:14:36.760 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:04.967 WARNING:scrape:Ignoring malparsed backtrace: ==00:00:15:16.869 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.872 85287== Warning: unimplemented fcntl command: 1036
, ==00:00:15:16.938 85753== Warning: unimplemented fcntl command: 1036

2023-07-27T04:33:05.242 WARNING:scrape:Misunderstood line: 
2023-07-27T04:33:05.724 WARNING:scrape:Misunderstood line: 
2023-07-27T04:33:06.362 WARNING:scrape:Misunderstood line: 
2023-07-27T04:33:06.501 WARNING:scrape:Misunderstood line: 
2023-07-27T04:33:06.697 INFO:scrape:Found 6 distinct failure reasons
2023-07-27T04:33:06.698 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c1444501ab7918ce42bdc26b9d860ad26e34dd69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2023-07-27T04:33:06.698 INFO:scrape:1 jobs: ['7353143']
2023-07-27T04:33:06.699 INFO:scrape:suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/e2e}']
2023-07-27T04:33:06.699 INFO:scrape:
2023-07-27T04:33:06.699 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi163.front.sepia.ceph.com: ['type=AVC msg=audit(1690409663.362:20749): avc:  denied  { ioctl } for  pid=112684 comm="iptables" path="/var/lib/containers/storage/overlay/7ad4d8c21ec4ddefb3e161835936fbe86a57301cf46fa1920705f578cb08f9a4/merged" dev="overlay" ino=3408726 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
2023-07-27T04:33:06.699 INFO:scrape:1 jobs: ['7353140']
2023-07-27T04:33:06.700 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_cephadm}']
2023-07-27T04:33:06.701 INFO:scrape:
2023-07-27T04:33:06.701 INFO:scrape:Dead: None
2023-07-27T04:33:06.702 INFO:scrape:4 jobs: ['7353097', '7353072', '7353069', '7353061']
2023-07-27T04:33:06.703 INFO:scrape:
2023-07-27T04:33:06.703 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh
2023-07-27T04:33:06.703 INFO:scrape:2 jobs: ['7353145', '7353055']
2023-07-27T04:33:06.704 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2023-07-27T04:33:06.704 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}']
2023-07-27T04:33:06.705 INFO:scrape:
2023-07-27T04:33:06.705 INFO:scrape:Valgrind: mon (Leak_StillReachable)
2023-07-27T04:33:06.706 INFO:scrape:4 jobs: ['7353094', '7353053', '7353125', '7353091']
2023-07-27T04:33:06.706 INFO:scrape:suites intersection: []
2023-07-27T04:33:06.707 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'validater/valgrind}']
2023-07-27T04:33:06.707 INFO:scrape:
2023-07-27T04:33:06.708 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats'
2023-07-27T04:33:06.708 INFO:scrape:1 jobs: ['7353076']
2023-07-27T04:33:06.709 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{centos_latest}}']
2023-07-27T04:33:06.709 INFO:scrape: