2023-05-25T16:57:32.849 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-25T16:57:32.850 INFO:teuthology.results:seed: '7907' 2023-05-25T17:18:49.778 INFO:root:teuthology version: 0.0.1.dev109+ge5a3f9b 2023-05-25T17:18:49.852 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-25T21:23:08.218 INFO:teuthology.results:Tests finished! gathering results... 2023-05-25T21:23:09.655 INFO:scrape:Found 20 jobs 2023-05-25T21:23:09.993 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-4146-gd5af2802 (d5af280272eba880f0e885bddd69eba3890b5eaa) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1004fc2ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5584742e60ca] 2023-05-25T21:23:09.994 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:38:11.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~39s , 2023-05-25T18:38:11.734 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:38:11.854+0000 7fe52d417700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:09.995 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:39:12.021 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:39:12.141+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:39:12.147+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:09.996 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:40:12.494 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:40:12.616+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:40:12.622+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:09.997 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:41:13.153 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:41:13.273+0000 7fb504af1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:41:13.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2023-05-25T21:23:09.998 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:42:13.817 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:42:13.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s , 2023-05-25T18:42:13.937+0000 7f5231b8c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:09.999 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:09.999 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T19:34:49.858 DEBUG:tasks.ceph.osd.3:waiting for process to exit , 2023-05-25T19:34:49.859 INFO:teuthology.orchestra.run:waiting for 300 , 2023-05-25T19:34:49.963 INFO:tasks.ceph.osd.3:Stopped 2023-05-25T21:23:10.000 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.144737.log.gz , /var/log/ceph/ceph-client.admin.163257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131768.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.163257.log.gz 2023-05-25T21:23:10.000 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.148368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139314.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148368.log.gz , /var/log/ceph/ceph-client.admin.165980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152902.log 2023-05-25T21:23:10.000 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.156757.log.gz , /var/log/ceph/ceph-client.admin.161769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161561.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.161769.log.gz 2023-05-25T21:23:10.001 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.142085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161463.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.142085.log.gz , /var/log/ceph/ceph-client.admin.134538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157961.log 2023-05-25T21:23:10.001 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.158123.log.gz , /var/log/ceph/ceph-client.admin.162171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154330.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.162171.log.gz 2023-05-25T21:23:10.002 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.146617.log.gz , /var/log/ceph/ceph-client.admin.148924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154668.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148924.log.gz 2023-05-25T21:23:10.002 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.165432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136759.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.165432.log.gz , /var/log/ceph/ceph-client.admin.156862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147535.log 2023-05-25T21:23:10.003 WARNING:scrape:Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.136484.log.gz , /var/log/ceph/ceph-client.admin.138662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164795.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.138662.log.gz 2023-05-25T21:23:10.003 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.137251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133422.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.137251.log.gz , /var/log/ceph/ceph-client.admin.163453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149036.log 2023-05-25T21:23:10.004 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.133987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134321.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.133987.log.gz , /var/log/ceph/ceph-client.admin.162759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148980.log 2023-05-25T21:23:10.004 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.157702.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.157702.log.gz , /var/log/ceph/ceph-client.admin.159005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156701.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.159005.log.gz 2023-05-25T21:23:10.004 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.142514.log.gz , /var/log/ceph/ceph-client.admin.152685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144198.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.152685.log.gz 2023-05-25T21:23:10.005 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.145965.log.gz , /var/log/ceph/ceph-client.admin.147584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160745.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.147584.log.gz 2023-05-25T21:23:10.005 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:10.006 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise 2023-05-25T21:23:10.052 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-4146-gd5af2802 (d5af280272eba880f0e885bddd69eba3890b5eaa) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1004fc2ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5584742e60ca] 2023-05-25T21:23:10.053 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:38:11.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~39s , 2023-05-25T18:38:11.734 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:38:11.854+0000 7fe52d417700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.054 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:39:12.021 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:39:12.141+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:39:12.147+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.055 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:40:12.494 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:40:12.616+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:40:12.622+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.056 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:41:13.153 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:41:13.273+0000 7fb504af1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:41:13.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2023-05-25T21:23:10.057 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:42:13.817 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:42:13.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s , 2023-05-25T18:42:13.937+0000 7f5231b8c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.057 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:10.058 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T19:34:49.858 DEBUG:tasks.ceph.osd.3:waiting for process to exit , 2023-05-25T19:34:49.859 INFO:teuthology.orchestra.run:waiting for 300 , 2023-05-25T19:34:49.963 INFO:tasks.ceph.osd.3:Stopped 2023-05-25T21:23:10.058 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.144737.log.gz , /var/log/ceph/ceph-client.admin.163257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131768.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.163257.log.gz 2023-05-25T21:23:10.059 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.148368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139314.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148368.log.gz , /var/log/ceph/ceph-client.admin.165980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152902.log 2023-05-25T21:23:10.059 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.156757.log.gz , /var/log/ceph/ceph-client.admin.161769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161561.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.161769.log.gz 2023-05-25T21:23:10.059 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.142085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161463.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.142085.log.gz , /var/log/ceph/ceph-client.admin.134538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157961.log 2023-05-25T21:23:10.060 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.158123.log.gz , /var/log/ceph/ceph-client.admin.162171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154330.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.162171.log.gz 2023-05-25T21:23:10.060 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.146617.log.gz , /var/log/ceph/ceph-client.admin.148924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154668.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148924.log.gz 2023-05-25T21:23:10.061 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.165432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136759.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.165432.log.gz , /var/log/ceph/ceph-client.admin.156862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147535.log 2023-05-25T21:23:10.061 WARNING:scrape:Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.136484.log.gz , /var/log/ceph/ceph-client.admin.138662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164795.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.138662.log.gz 2023-05-25T21:23:10.062 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.137251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133422.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.137251.log.gz , /var/log/ceph/ceph-client.admin.163453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149036.log 2023-05-25T21:23:10.062 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.133987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134321.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.133987.log.gz , /var/log/ceph/ceph-client.admin.162759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148980.log 2023-05-25T21:23:10.063 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.157702.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.157702.log.gz , /var/log/ceph/ceph-client.admin.159005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156701.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.159005.log.gz 2023-05-25T21:23:10.063 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.142514.log.gz , /var/log/ceph/ceph-client.admin.152685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144198.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.152685.log.gz 2023-05-25T21:23:10.063 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.145965.log.gz , /var/log/ceph/ceph-client.admin.147584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160745.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.147584.log.gz 2023-05-25T21:23:10.064 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:10.064 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise 2023-05-25T21:23:10.111 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-4146-gd5af2802 (d5af280272eba880f0e885bddd69eba3890b5eaa) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1004fc2ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5584742e60ca] 2023-05-25T21:23:10.113 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:38:11.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~39s , 2023-05-25T18:38:11.734 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:38:11.854+0000 7fe52d417700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.114 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:39:12.021 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:39:12.141+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:39:12.147+0000 7f1b5cd17700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.115 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:40:12.494 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:40:12.616+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:40:12.622+0000 7fdbccbb5700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:41:13.153 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:41:13.273+0000 7fb504af1700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-05-25T18:41:13.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2023-05-25T21:23:10.116 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T18:42:13.817 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json , 2023-05-25T18:42:13.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s , 2023-05-25T18:42:13.937+0000 7f5231b8c700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-05-25T21:23:10.117 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:10.117 WARNING:scrape:Ignoring malparsed backtrace: 2023-05-25T19:34:49.858 DEBUG:tasks.ceph.osd.3:waiting for process to exit , 2023-05-25T19:34:49.859 INFO:teuthology.orchestra.run:waiting for 300 , 2023-05-25T19:34:49.963 INFO:tasks.ceph.osd.3:Stopped 2023-05-25T21:23:10.118 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.144737.log.gz , /var/log/ceph/ceph-client.admin.163257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131768.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.163257.log.gz 2023-05-25T21:23:10.118 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.148368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139314.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148368.log.gz , /var/log/ceph/ceph-client.admin.165980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152902.log 2023-05-25T21:23:10.119 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.156757.log.gz , /var/log/ceph/ceph-client.admin.161769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161561.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.161769.log.gz 2023-05-25T21:23:10.119 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.142085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161463.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.142085.log.gz , /var/log/ceph/ceph-client.admin.134538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157961.log 2023-05-25T21:23:10.120 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.158123.log.gz , /var/log/ceph/ceph-client.admin.162171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154330.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.162171.log.gz 2023-05-25T21:23:10.120 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.146617.log.gz , /var/log/ceph/ceph-client.admin.148924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154668.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.148924.log.gz 2023-05-25T21:23:10.120 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.165432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136759.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.165432.log.gz , /var/log/ceph/ceph-client.admin.156862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147535.log 2023-05-25T21:23:10.121 WARNING:scrape:Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.136484.log.gz , /var/log/ceph/ceph-client.admin.138662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164795.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.138662.log.gz 2023-05-25T21:23:10.121 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.137251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133422.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.137251.log.gz , /var/log/ceph/ceph-client.admin.163453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149036.log 2023-05-25T21:23:10.122 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.133987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134321.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.133987.log.gz , /var/log/ceph/ceph-client.admin.162759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148980.log 2023-05-25T21:23:10.122 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.157702.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.157702.log.gz , /var/log/ceph/ceph-client.admin.159005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156701.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.159005.log.gz 2023-05-25T21:23:10.123 WARNING:scrape:Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.142514.log.gz , /var/log/ceph/ceph-client.admin.152685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144198.log , 49.6% -- replaced with /var/log/ceph/ceph-client.admin.152685.log.gz 2023-05-25T21:23:10.123 WARNING:scrape:Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.145965.log.gz , /var/log/ceph/ceph-client.admin.147584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160745.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.147584.log.gz 2023-05-25T21:23:10.124 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/teuthology/orchestra/run.py", line 161, in wait 2023-05-25T21:23:10.124 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_e5a3f9b5537d0e955855bcef418326f19d40643a/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise 2023-05-25T21:23:10.313 INFO:scrape:Found 3 distinct failure reasons 2023-05-25T21:23:10.314 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh --eval-attr \ 2023-05-25T21:23:10.314 INFO:scrape:2 jobs: ['7286640', '7286643'] 2023-05-25T21:23:10.315 INFO:scrape:suites intersection: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-05-25T21:23:10.315 INFO:scrape:suites union: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}', 'tasks/rbd_python_api_tests}'] 2023-05-25T21:23:10.315 INFO:scrape: 2023-05-25T21:23:10.316 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2023-05-25T21:23:10.316 INFO:scrape:1 jobs: ['7286638'] 2023-05-25T21:23:10.316 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2023-05-25T21:23:10.317 INFO:scrape: 2023-05-25T21:23:10.317 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-05-25T21:23:10.317 INFO:scrape:2 jobs: ['7286629', '7286626'] 2023-05-25T21:23:10.317 INFO:scrape:suites intersection: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-05-25T21:23:10.318 INFO:scrape:suites union: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2023-05-25T21:23:10.318 INFO:scrape: