2022-12-21T14:31:58.542 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-21T14:31:58.542 INFO:teuthology.results:subset: '1/2'
2022-12-21T14:31:58.542 INFO:teuthology.results:seed: '8303'
2022-12-21T18:43:54.379 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2022-12-21T18:43:54.419 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-21T19:35:58.820 INFO:teuthology.results:Tests finished! gathering results...
2022-12-21T19:36:04.147 INFO:scrape:Found 72 jobs
2022-12-21T19:36:04.148 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/dang-2022-12-21_09:11:10-rgw-wip-dang-bucket-policy-distro-default-smithi/7123573/teuthology.log
2022-12-21T19:36:07.862 WARNING:scrape:Not-understood crash indication 2022-12-21T16:01:10.143 INFO:tasks.rgw.c1.client.0.smithi109.stderr:daemon-helper: command crashed with signal 6
2022-12-21T19:36:07.863 WARNING:scrape:Not-understood crash indication 2022-12-21T17:09:45.344 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi109.stderr: INFO: daemon-helper: command crashed with signal 6
2022-12-21T19:36:07.863 WARNING:scrape:Not-understood crash indication 
2022-12-21T19:36:08.076 WARNING:scrape:Not-understood crash indication 2022-12-21T16:01:10.143 INFO:tasks.rgw.c1.client.0.smithi109.stderr:daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.076 WARNING:scrape:Not-understood crash indication 2022-12-21T17:09:45.344 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi109.stderr: INFO: daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.076 WARNING:scrape:Not-understood crash indication 
2022-12-21T19:36:08.275 WARNING:scrape:Not-understood crash indication 2022-12-21T16:01:10.143 INFO:tasks.rgw.c1.client.0.smithi109.stderr:daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.276 WARNING:scrape:Not-understood crash indication 2022-12-21T17:09:45.344 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi109.stderr: INFO: daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.276 WARNING:scrape:Not-understood crash indication 
2022-12-21T19:36:08.487 WARNING:scrape:Not-understood crash indication 2022-12-21T16:01:10.143 INFO:tasks.rgw.c1.client.0.smithi109.stderr:daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.488 WARNING:scrape:Not-understood crash indication 2022-12-21T17:09:45.344 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi109.stderr: INFO: daemon-helper: command crashed with signal 6
2022-12-21T19:36:08.488 WARNING:scrape:Not-understood crash indication 
2022-12-21T19:36:08.488 INFO:scrape:Found 5 distinct failure reasons
2022-12-21T19:36:08.489 INFO:scrape:Dead: None
2022-12-21T19:36:08.489 INFO:scrape:2 jobs: ['7123573', '7123586']
2022-12-21T19:36:08.489 INFO:scrape:
2022-12-21T19:36:08.490 INFO:scrape:Failure: Command failed on smithi016 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'
2022-12-21T19:36:08.490 INFO:scrape:1 jobs: ['7123546']
2022-12-21T19:36:08.490 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2022-12-21T19:36:08.490 INFO:scrape:
2022-12-21T19:36:08.490 INFO:scrape:Failure: {'smithi093.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:04.136473', 'end': '2022-12-21 14:44:14.835805', 'invocation': {'module_args': {'_raw_params': 'cpan  Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-21 14:44:10.699332', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/authors/01mailrc.txt.gz\nReading '/home/ubuntu/.cpan/sources/authors/01mailrc.txt.gz'\n............................................................................DONE\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/modules/02packages.details.txt.gz\nReading '/home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz'\n  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nWarning: This index file is 2504 days old.\n  Please check the host you chose as your CPAN mirror for staleness.\n  I'll continue but problems seem likely to happen.\x07\n............................................................................DONE\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/modules/03modlist.data.gz\nReading '/home/ubuntu/.cpan/sources/modules/03modlist.data.gz'\nDONE\nWriting /home/ubuntu/.cpan/Metadata\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", '  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/authors/01mailrc.txt.gz', "Reading '/home/ubuntu/.cpan/sources/authors/01mailrc.txt.gz'", '............................................................................DONE', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/modules/02packages.details.txt.gz', "Reading '/home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz'", '  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', 'Warning: This index file is 2504 days old.', '  Please check the host you chose as your CPAN mirror for staleness.', "  I'll continue but problems seem likely to happen.\x07", '............................................................................DONE', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/modules/03modlist.data.gz', "Reading '/home/ubuntu/.cpan/sources/modules/03modlist.data.gz'", 'DONE', 'Writing /home/ubuntu/.cpan/Metadata', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}, 'smithi149.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:04.271390', 'end': '2022-12-21 14:44:15.150693', 'invocation': {'module_args': {'_raw_params': 'cpan  Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-21 14:44:10.879303', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/authors/01mailrc.txt.gz\nReading '/home/ubuntu/.cpan/sources/authors/01mailrc.txt.gz'\n............................................................................DONE\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/modules/02packages.details.txt.gz\nReading '/home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz'\n  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nWarning: This index file is 2504 days old.\n  Please check the host you chose as your CPAN mirror for staleness.\n  I'll continue but problems seem likely to happen.\x07\n............................................................................DONE\nFetching with LWP:\nhttp://apt-mirror.sepia.ceph.com/CPAN/modules/03modlist.data.gz\nReading '/home/ubuntu/.cpan/sources/modules/03modlist.data.gz'\nDONE\nWriting /home/ubuntu/.cpan/Metadata\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", '  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/authors/01mailrc.txt.gz', "Reading '/home/ubuntu/.cpan/sources/authors/01mailrc.txt.gz'", '............................................................................DONE', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/modules/02packages.details.txt.gz', "Reading '/home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz'", '  Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', 'Warning: This index file is 2504 days old.', '  Please check the host you chose as your CPAN mirror for staleness.', "  I'll continue but problems seem likely to happen.\x07", '............................................................................DONE', 'Fetching with LWP:', 'http://apt-mirror.sepia.ceph.com/CPAN/modules/03modlist.data.gz', "Reading '/home/ubuntu/.cpan/sources/modules/03modlist.data.gz'", 'DONE', 'Writing /home/ubuntu/.cpan/Metadata', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}}
2022-12-21T19:36:08.491 INFO:scrape:2 jobs: ['7123538', '7123577']
2022-12-21T19:36:08.491 INFO:scrape:suites intersection: ['overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}']
2022-12-21T19:36:08.491 INFO:scrape:suites union: ['hadoop/default', 'hadoop/v32', 'overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}']
2022-12-21T19:36:08.491 INFO:scrape:
2022-12-21T19:36:08.491 INFO:scrape:Crash: Command failed on smithi062 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph'
2022-12-21T19:36:08.492 INFO:scrape:ceph version 18.0.0-1503-g0ac32cff (0ac32cffb257acb04ad98c95abcec3994ddcc249) reef (dev)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fc3c6e6acf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc3c964edf5]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a7f61) [0x7fc3c964ef61]
 6: ceph-mon(+0x3084e6) [0x5592483ee4e6]
 7: (Monitor::~Monitor()+0xd) [0x5592483ee52d]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-12-21T19:36:08.492 INFO:scrape:1 jobs: ['7123542']
2022-12-21T19:36:08.492 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'overrides', 'rgw/notifications/{beast', 'supported-all-distro$/{centos_8}', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}']
2022-12-21T19:36:08.492 INFO:scrape:
2022-12-21T19:36:08.492 INFO:scrape:Failure: rgw multisite test failures
2022-12-21T19:36:08.493 INFO:scrape:2 jobs: ['7123578', '7123541']
2022-12-21T19:36:08.493 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2022-12-21T19:36:08.493 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'tasks/test_multi}']
2022-12-21T19:36:08.493 INFO:scrape: