2023-03-14T21:34:07.819 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T21:34:07.820 INFO:teuthology.results:subset: '111/120000' 2023-03-14T21:34:07.820 INFO:teuthology.results:seed: '9244' 2023-03-15T03:10:25.969 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-15T03:10:25.997 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-15T09:53:03.611 INFO:teuthology.results:Tests finished! gathering results... 2023-03-15T09:53:27.749 INFO:scrape:Found 320 jobs 2023-03-15T09:53:30.682 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:30.684 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:30.684 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:30.685 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:30.685 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:30.764 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:30.766 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:30.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:30.767 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:30.768 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:30.848 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:30.849 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:30.850 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:30.850 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:30.851 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:30.927 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:30.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:30.929 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:30.930 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:30.930 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.007 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.008 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.009 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.009 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.010 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.089 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.090 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.091 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.092 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.172 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.173 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.174 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.174 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.175 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.252 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.254 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.254 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.255 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.255 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.331 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.332 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.333 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.334 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.409 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.411 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.412 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.490 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.5-1340-g714f8ff9 (714f8ff94ab1a8a5b10ea54247535614e53b7234) quincy (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fce8fd69483] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x26a649) [0x7fce8fd69649] 2023-03-15T09:53:31.492 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:19:52.838 INFO:teuthology.orchestra.run.smithi140.stdout:{ , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2023-03-15T00:19:52.839 INFO:teuthology.orchestra.run.smithi140.stdout:} 2023-03-15T09:53:31.492 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:21:15.791 INFO:teuthology.orchestra.run:waiting for 300 , 2023-03-15T00:21:15.848 INFO:tasks.ceph.osd.3:Stopped , 2023-03-15T00:21:15.849 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-03-15T09:53:31.493 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-15T00:22:26.010 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2023-03-15T00:22:26.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-15T09:53:31.493 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.590 INFO:scrape:Found 10 distinct failure reasons 2023-03-15T09:53:31.591 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 2023-03-15T09:53:31.592 INFO:scrape:2 jobs: ['7207582', '7207883'] 2023-03-15T09:53:31.592 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}'] 2023-03-15T09:53:31.592 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-stupid}', 'tasks/e2e}'] 2023-03-15T09:53:31.592 INFO:scrape: 2023-03-15T09:53:31.593 INFO:scrape:Failure: Command failed on smithi114 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:714f8ff94ab1a8a5b10ea54247535614e53b7234 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec612c9e-c2c1-11ed-9afb-001a4aab830c -- ceph osd stat -f json' 2023-03-15T09:53:31.593 INFO:scrape:1 jobs: ['7207563'] 2023-03-15T09:53:31.593 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/repave-all}', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0'] 2023-03-15T09:53:31.594 INFO:scrape: 2023-03-15T09:53:31.594 INFO:scrape:Failure: Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) 2023-03-15T09:53:31.594 INFO:scrape:2 jobs: ['7207836', '7207549'] 2023-03-15T09:53:31.595 INFO:scrape:suites intersection: ['task/test_nfs}'] 2023-03-15T09:53:31.595 INFO:scrape:suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8', 'task/test_nfs}'] 2023-03-15T09:53:31.595 INFO:scrape: 2023-03-15T09:53:31.595 INFO:scrape:Failure: Command failed on smithi139 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' 2023-03-15T09:53:31.596 INFO:scrape:4 jobs: ['7207759', '7207885', '7207585', '7207471'] 2023-03-15T09:53:31.596 INFO:scrape:suites intersection: ['0-kubeadm', '0-nvme-loop', '1-rook', '3-final', 'k8s/1.21', 'rados/rook/smoke/{0-distro/ubuntu_20.04'] 2023-03-15T09:53:31.596 INFO:scrape:suites union: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '2-workload/radosbench', '3-final', 'cluster/1-node', 'cluster/3-node', 'k8s/1.21', 'net/calico', 'net/flannel', 'net/host', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}', 'rook/master}'] 2023-03-15T09:53:31.597 INFO:scrape: 2023-03-15T09:53:31.597 INFO:scrape:Dead: b'2023-03-15T09:51:02.714 DEBUG:teuthology.exit:Finished running handlers' 2023-03-15T09:53:31.597 INFO:scrape:1 jobs: ['7207417'] 2023-03-15T09:53:31.597 INFO:scrape:suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/pool-access', 'supported-random-distro$/{ubuntu_latest}}'] 2023-03-15T09:53:31.598 INFO:scrape: 2023-03-15T09:53:31.598 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi033.front.sepia.ceph.com: ['type=AVC msg=audit(1678844700.936:19535): avc: denied { ioctl } for pid=124960 comm="iptables" path="/var/lib/containers/storage/overlay/68468036b94a513311c0ee11a4f31e77322398583544821691a7c7aea95056f4/merged" dev="overlay" ino=4067387 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 2023-03-15T09:53:31.598 INFO:scrape:1 jobs: ['7207719'] 2023-03-15T09:53:31.599 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'task/test_cephadm}'] 2023-03-15T09:53:31.599 INFO:scrape: 2023-03-15T09:53:31.599 INFO:scrape:Failure: Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' 2023-03-15T09:53:31.599 INFO:scrape:1 jobs: ['7207550'] 2023-03-15T09:53:31.600 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/test_envlibrados_for_rocksdb', 'supported-random-distro$/{rhel_8}}'] 2023-03-15T09:53:31.600 INFO:scrape: 2023-03-15T09:53:31.600 INFO:scrape:Failure: Command failed on smithi067 with status 127: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' 2023-03-15T09:53:31.601 INFO:scrape:1 jobs: ['7207727'] 2023-03-15T09:53:31.601 INFO:scrape:suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{rhel_8}', 'tasks/rados_workunit_loadgen_mostlyread}'] 2023-03-15T09:53:31.601 INFO:scrape: 2023-03-15T09:53:31.601 INFO:scrape:Failure: Command failed (workunit test misc/mclock-config.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh' 2023-03-15T09:53:31.602 INFO:scrape:1 jobs: ['7207723'] 2023-03-15T09:53:31.602 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/misc}'] 2023-03-15T09:53:31.602 INFO:scrape: 2023-03-15T09:53:31.602 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2023-03-15T09:53:31.603 INFO:scrape:1 jobs: ['7207616'] 2023-03-15T09:53:31.603 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{rhel_8}}'] 2023-03-15T09:53:31.603 INFO:scrape: 2023-03-15T09:53:31.614 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/ failed: 14 dead: 1 running: 0 waiting: 0 queued: 0 passed: 305 Fail ================================================================= [7207471] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} ----------------------------------------------------------------- time: 00:08:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207471/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=779cade3f87b40029424bfc3be57d02a Command failed on smithi016 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7207549] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} ----------------------------------------------------------------- time: 00:24:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207549/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207549/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9cf59320a564359a000d0c00be13619 Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7207550] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:13:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207550/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207550/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5179cd282074cd68ffd94c6ba70a841 Command failed (workunit test rados/test_envlibrados_for_rocksdb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rados/test_envlibrados_for_rocksdb.sh' [7207563] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:13:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207563/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207563/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53df38f345cf4a288241470a6921b130 Command failed on smithi114 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:714f8ff94ab1a8a5b10ea54247535614e53b7234 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec612c9e-c2c1-11ed-9afb-001a4aab830c -- ceph osd stat -f json' [7207582] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} tasks/e2e} ----------------------------------------------------------------- time: 00:18:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207582/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70b0dfa08d0f42f4b20b9ab6a5b2f2e0 Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7207585] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} ----------------------------------------------------------------- time: 00:06:27 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207585/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207585/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46bec9a21fef486cbbd67c29ef87f715 Command failed on smithi005 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7207616] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:17:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207616/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207616/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [7207719] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:17:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207719/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207719/ SELinux denials found on ubuntu@smithi033.front.sepia.ceph.com: ['type=AVC msg=audit(1678844700.936:19535): avc: denied { ioctl } for pid=124960 comm="iptables" path="/var/lib/containers/storage/overlay/68468036b94a51331 1c0ee11a4f31e77322398583544821691a7c7aea95056f4/merged" dev="overlay" ino=4067387 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] [7207723] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} ----------------------------------------------------------------- time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207723/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207723/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d9605bbc77b419bbd6e4bf9febf3724 Command failed (workunit test misc/mclock-config.sh) on smithi120 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh' [7207727] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207727/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3a9a3879e6d45c9bb3c87e9e82cf6a2 Command failed on smithi067 with status 127: 'sudo MALLOC_CHECK_=3 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' [7207759] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} ----------------------------------------------------------------- time: 00:06:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207759/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b14a61ed5dfd420cb87742143569e0e1 Command failed on smithi139 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' [7207836] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:24:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207836/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=623ccbe3db4a4ce6a551d950a3e27c88 Test failure: test_non_existent_cluster (tasks.cephfs.test_nfs.TestNFS) [7207883] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/e2e} ----------------------------------------------------------------- time: 00:18:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207883/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207883/ sentry: https://sentry.ceph.com/organizations/ceph/?query=103bb1366cd5413e9f167e59812b3e1b Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' [7207885] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} ----------------------------------------------------------------- time: 00:06:21 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207885/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207885/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a017b0c2da654ae5868320172e45282a Command failed on smithi110 with status 1: 'sudo systemctl enable --now kubelet && sudo kubeadm config images pull' Dead ================================================================= [7207417] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207417/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207417/ hit max job timeout Pass ================================================================= [7207397] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{rhel_8} tasks/workunits} time: 00:13:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207397/ [7207399] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:12:11 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207399/ [7207401] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:25:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207401/ [7207403] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_orch_cli_mon} time: 00:27:46 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207403/ [7207405] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 00:28:56 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207405/ [7207407] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{centos_8}} time: 00:37:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207407/ [7207409] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_osdmap_prune} time: 00:46:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207409/ [7207411] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:32:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207411/ [7207413] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} time: 00:34:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207413/ [7207415] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:29:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207415/ [7207419] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:20:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207419/ [7207421] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:13:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207421/ [7207423] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:16:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207423/ [7207425] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:11:00 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207425/ [7207427] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 01:02:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207427/ [7207429] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_adoption} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207429/ [7207431] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:32:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207431/ [7207433] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:19:18 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207433/ [7207435] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:34:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207435/ [7207437] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207437/ [7207439] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:24:40 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207439/ [7207441] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:11:12 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207441/ [7207443] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:22:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207443/ [7207445] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:16:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207445/ [7207447] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:53 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207447/ [7207449] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207449/ [7207451] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:37:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207451/ [7207453] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:17:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207453/ [7207455] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:16:27 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207455/ [7207457] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:22:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207457/ [7207459] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:30:52 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207459/ [7207461] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:14:52 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207461/ [7207463] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{filestore-xfs} tasks/dashboard} time: 01:12:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207463/ [7207465] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:15:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207465/ [7207467] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:14:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207467/ [7207469] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:15:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207469/ [7207473] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:18:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207473/ [7207475] rados/standalone/{supported-random-distro$/{rhel_8} workloads/c2c} time: 00:17:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207475/ [7207477] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:21:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207477/ [7207479] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:22:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207479/ [7207481] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} time: 00:20:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207481/ [7207483] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:22:46 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207483/ [7207485] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:30:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207485/ [7207487] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} time: 00:27:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207487/ [7207489] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207489/ [7207491] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:27:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207491/ [7207494] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207494/ [7207495] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207495/ [7207497] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207497/ [7207499] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:09:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207499/ [7207501] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:18:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207501/ [7207503] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:30:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207503/ [7207504] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} time: 00:13:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207504/ [7207505] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:32:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207505/ [7207506] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:31:01 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207506/ [7207507] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207507/ [7207508] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:20:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207508/ [7207509] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207509/ [7207510] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207510/ [7207511] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:52 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207511/ [7207512] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:15:28 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207512/ [7207513] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:13:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207513/ [7207514] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:22:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207514/ [7207515] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{centos_8}} time: 00:15:55 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207515/ [7207516] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:06:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207516/ [7207517] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:11:22 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207517/ [7207518] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:23:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207518/ [7207519] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207519/ [7207520] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{rhel_8} tasks/failover} time: 00:16:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207520/ [7207521] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits} time: 00:33:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207521/ [7207522] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207522/ [7207523] rados/standalone/{supported-random-distro$/{rhel_8} workloads/crush} time: 00:25:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207523/ [7207524] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:23:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207524/ [7207525] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:31:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207525/ [7207526] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:52:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207526/ [7207527] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207527/ [7207528] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:16:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207528/ [7207529] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:13:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207529/ [7207530] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:34:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207530/ [7207531] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_pids_limit} time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207531/ [7207532] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:27:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207532/ [7207533] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207533/ [7207534] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:18:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207534/ [7207535] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:14:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207535/ [7207536] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:36:45 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207536/ [7207537] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207537/ [7207538] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:30:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207538/ [7207539] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:46 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207539/ [7207540] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207540/ [7207541] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207541/ [7207542] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:01 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207542/ [7207543] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207543/ [7207544] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:11:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207544/ [7207545] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:25:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207545/ [7207546] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:12:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207546/ [7207547] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207547/ [7207548] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:22:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207548/ [7207551] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} time: 00:20:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207551/ [7207552] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/one workloads/snaps-few-objects} time: 00:38:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207552/ [7207553] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:28 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207553/ [7207555] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/insights} time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207555/ [7207557] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207557/ [7207559] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} time: 00:09:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207559/ [7207561] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:36:22 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207561/ [7207565] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:13:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207565/ [7207567] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 01:27:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207567/ [7207569] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 01:14:27 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207569/ [7207571] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:26 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207571/ [7207573] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:45:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207573/ [7207576] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:09:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207576/ [7207579] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:17:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207579/ [7207588] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:13:18 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207588/ [7207590] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:10:37 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207590/ [7207592] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:43:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207592/ [7207594] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:23:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207594/ [7207596] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli} time: 00:18:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207596/ [7207598] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:20:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207598/ [7207600] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:57 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207600/ [7207602] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} time: 02:26:22 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207602/ [7207604] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:24:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207604/ [7207606] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:55 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207606/ [7207608] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:18:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207608/ [7207610] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:15:34 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207610/ [7207612] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:17:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207612/ [7207614] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big} time: 00:31:09 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207614/ [7207618] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207618/ [7207620] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:16:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207620/ [7207622] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} time: 00:30:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207622/ [7207624] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:09:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207624/ [7207626] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} time: 00:19:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207626/ [7207628] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207628/ [7207630] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:28:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207630/ [7207632] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:26:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207632/ [7207634] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:09:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207634/ [7207636] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:28:28 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207636/ [7207638] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:33:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207638/ [7207640] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:18:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207640/ [7207642] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207642/ [7207644] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:22:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207644/ [7207646] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:35:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207646/ [7207648] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:23:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207648/ [7207650] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:24:24 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207650/ [7207652] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:24:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207652/ [7207655] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:11:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207655/ [7207658] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:34:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207658/ [7207661] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:30:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207661/ [7207664] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:19:39 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207664/ [7207667] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:46 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207667/ [7207670] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:22:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207670/ [7207673] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_adoption} time: 00:10:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207673/ [7207676] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:24:11 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207676/ [7207679] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:28:01 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207679/ [7207682] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:21:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207682/ [7207685] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:12:00 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207685/ [7207688] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207688/ [7207691] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:21:18 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207691/ [7207694] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:21:40 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207694/ [7207697] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:10:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207697/ [7207700] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} time: 00:28:01 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207700/ [7207703] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207703/ [7207705] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_5925} time: 00:14:57 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207705/ [7207707] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207707/ [7207709] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:36 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207709/ [7207711] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:19:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207711/ [7207713] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/progress} time: 00:26:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207713/ [7207715] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:15:30 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207715/ [7207717] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:12:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207717/ [7207721] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:33:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207721/ [7207725] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:13:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207725/ [7207729] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:10:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207729/ [7207731] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:24:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207731/ [7207733] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} time: 00:10:57 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207733/ [7207735] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207735/ [7207737] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207737/ [7207739] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:32:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207739/ [7207741] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207741/ [7207743] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207743/ [7207745] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207745/ [7207747] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:27:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207747/ [7207749] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207749/ [7207751] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:10:59 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207751/ [7207753] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:10:59 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207753/ [7207755] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:31:26 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207755/ [7207757] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} time: 01:09:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207757/ [7207761] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:21:09 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207761/ [7207763] rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:20:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207763/ [7207765] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_cephadm_repos} time: 00:08:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207765/ [7207767] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 01:04:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207767/ [7207769] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} time: 02:19:49 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207769/ [7207771] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207771/ [7207773] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_api_tests} time: 00:30:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207773/ [7207775] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:55 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207775/ [7207777] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:34:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207777/ [7207779] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/basic 3-final} time: 00:14:18 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207779/ [7207782] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/readwrite} time: 00:14:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207782/ [7207784] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207784/ [7207786] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207786/ [7207788] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 01:02:24 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207788/ [7207790] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/prometheus} time: 00:15:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207790/ [7207792] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:31 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207792/ [7207794] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207794/ [7207796] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{rhel_8}} time: 00:17:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207796/ [7207798] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:36:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207798/ [7207800] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:27:32 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207800/ [7207802] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:16:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207802/ [7207804] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} time: 01:05:37 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207804/ [7207806] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:08:09 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207806/ [7207808] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:17:39 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207808/ [7207809] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207809/ [7207812] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench} time: 01:24:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207812/ [7207814] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 01:03:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207814/ [7207815] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207815/ [7207817] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207817/ [7207820] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/on mon_election/connectivity task/test_iscsi_pids_limit} time: 00:18:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207820/ [7207821] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:16:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207821/ [7207823] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/repair_test} time: 00:18:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207823/ [7207825] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect} time: 00:13:53 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207825/ [7207828] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207828/ [7207830] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207830/ [7207832] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:53:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207832/ [7207834] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:57:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207834/ [7207838] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207838/ [7207840] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:29:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207840/ [7207842] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:14:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207842/ [7207844] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207844/ [7207846] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:14:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207846/ [7207847] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:12:11 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207847/ [7207848] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} time: 00:27:15 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207848/ [7207849] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:12:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207849/ [7207850] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{rhel_8} tasks/workunits} time: 00:13:21 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207850/ [7207851] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:24:24 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207851/ [7207852] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:16:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207852/ [7207853] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:16 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207853/ [7207854] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:56 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207854/ [7207855] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:12:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207855/ [7207856] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:08:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207856/ [7207857] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill} time: 03:31:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207857/ [7207858] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:22:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207858/ [7207859] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207859/ [7207860] rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_orch_cli} time: 00:19:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207860/ [7207861] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:12:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207861/ [7207862] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:16:50 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207862/ [7207863] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:19:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207863/ [7207864] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207864/ [7207865] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:12:28 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207865/ [7207866] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:12:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207866/ [7207867] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:24:53 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207867/ [7207868] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207868/ [7207869] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:33:09 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207869/ [7207870] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:24:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207870/ [7207871] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} time: 00:55:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207871/ [7207872] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} time: 00:31:56 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207872/ [7207873] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207873/ [7207874] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:16:00 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207874/ [7207875] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207875/ [7207876] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:10:59 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207876/ [7207877] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_8}} time: 02:13:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207877/ [7207878] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:46:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207878/ [7207879] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:25:36 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207879/ [7207880] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207880/ [7207881] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} time: 00:25:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207881/ [7207882] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207882/ [7207884] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{rhel_8} tasks/crash} time: 00:15:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207884/ [7207886] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:21 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207886/ [7207887] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/libcephsqlite} time: 00:18:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207887/ [7207888] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects} time: 00:26:39 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207888/ [7207889] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:29:11 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207889/ [7207890] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207890/ [7207891] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:17:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207891/ [7207892] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:55 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207892/ [7207893] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:13:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207893/ [7207894] rados/standalone/{supported-random-distro$/{centos_8} workloads/osd} time: 02:55:28 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207894/ [7207895] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:29:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207895/ [7207896] rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} time: 00:20:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207896/ [7207897] rados/singleton/{all/mon-config mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:11:03 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207897/ [7207898] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207898/ [7207899] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:11:22 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207899/ [7207900] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207900/ [7207901] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep} time: 00:20:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207901/ [7207902] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:56 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207902/ [7207903] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:11:48 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207903/ [7207904] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207904/ [7207905] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/snaps-few-objects} time: 00:32:55 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207905/ [7207906] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:24:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207906/ [7207907] rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{centos_8}} time: 02:13:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207907/ [7207908] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:32:12 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207908/ [7207909] rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_adoption} time: 00:12:24 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207909/ [7207910] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:45 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207910/ [7207911] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} time: 00:28:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207911/ [7207912] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:28:15 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207912/ [7207913] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:17:53 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207913/ [7207914] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:38:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207914/ [7207915] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207915/ [7207916] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207916/ [7207917] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:18:26 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207917/ [7207918] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} supported-random-distro$/{rhel_8} tasks/failover} time: 00:17:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207918/ [7207919] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:28:05 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207919/ [7207920] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:32:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207920/ [7207921] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:10:46 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207921/ [7207922] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207922/ [7207923] rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_3.0 agent/off mon_election/classic task/test_cephadm} time: 00:19:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207923/ [7207924] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:26:47 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207924/ [7207925] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:11:39 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207925/ [7207926] rados/singleton/{all/peer mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:09:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207926/ [7207927] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:18:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_21:22:48-rados-quincy-release-distro-default-smithi/7207927/