2023-07-14T18:51:59.404 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e
2023-07-14T18:51:59.405 INFO:teuthology.results:seed: '2137'
2023-07-14T19:43:47.090 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e
2023-07-14T19:43:47.115 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-14T22:35:06.174 INFO:teuthology.results:Tests finished! gathering results...
2023-07-14T22:35:15.388 INFO:scrape:Found 70 jobs
2023-07-14T22:35:15.621 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/cbodley-2023-07-14_16:57:08-rgw-main-distro-default-smithi/7338068/teuthology.log
2023-07-14T22:35:16.259 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/cbodley-2023-07-14_16:57:08-rgw-main-distro-default-smithi/7338097/teuthology.log matches
2023-07-14T22:35:16.260 WARNING:scrape:Misunderstood line: 
2023-07-14T22:35:17.617 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/cbodley-2023-07-14_16:57:08-rgw-main-distro-default-smithi/7338076/teuthology.log matches
2023-07-14T22:35:17.618 WARNING:scrape:Misunderstood line: 
2023-07-14T22:35:17.621 INFO:scrape:Found 6 distinct failure reasons
2023-07-14T22:35:17.622 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi083 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'"
2023-07-14T22:35:17.622 INFO:scrape:2 jobs: ['7338061', '7338069']
2023-07-14T22:35:17.622 INFO:scrape:suites intersection: ['overrides', 's3tests-branch']
2023-07-14T22:35:17.623 INFO:scrape:suites union: ['first', 'ignore-pg-availability', 'objectstore', 'overrides', 'pool-type', 'rgw/dbstore/{cluster', 'rgw/sts/{cluster', 'rgw_frontend/beast', 's3tests-branch', 'ststests}}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/rgw_s3tests}', 'tasks/{0-install']
2023-07-14T22:35:17.623 INFO:scrape:
2023-07-14T22:35:17.623 INFO:scrape:Dead: None
2023-07-14T22:35:17.624 INFO:scrape:14 jobs: ['7338068', '7338114', '7338111', '7338123', '7338116', '7338081', '7338119', '7338124', '7338120', '7338125', '7338115', '7338121', '7338084', '7338122']
2023-07-14T22:35:17.625 INFO:scrape:
2023-07-14T22:35:17.625 INFO:scrape:Failure: rgw multisite test failures
2023-07-14T22:35:17.625 INFO:scrape:2 jobs: ['7338065', '7338101']
2023-07-14T22:35:17.626 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-07-14T22:35:17.626 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-07-14T22:35:17.626 INFO:scrape:
2023-07-14T22:35:17.627 INFO:scrape:Failure: smithi045.front.sepia.ceph.com:  changed: false  msg: All items completed  results:  - _ansible_item_label:      key: vg_nvme      value:        pvs: /dev/nvme0n1    _ansible_no_log: false    ansible_loop_var: item    changed: false    err: "  /dev/vg_nvme: already exists in filesystem\n  Run `vgcreate --help' for\      \ more information.\n"    failed: true    invocation:      module_args:        force: false        pesize: '4'        pv_options: ''        pvresize: false        pvs:        - /dev/nvme0n1        state: present        vg: vg_nvme        vg_options: ''    item:      key: vg_nvme      value:        pvs: /dev/nvme0n1    msg: Creating volume group 'vg_nvme' failed    rc: 3smithi155.front.sepia.ceph.com:  changed: false  msg: All items completed  results:  - _ansible_item_label:      key: vg_nvme      value:        pvs: /dev/nvme0n1    _ansible_no_log: false    ansible_loop_var: item    changed: false    err: "  /dev/vg_nvme: already exists in filesystem\n  Run `vgcreate --help' for\      \ more information.\n"    failed: true    invocation:      module_args:        force: false        pesize: '4'        pv_options: ''        pvresize: false        pvs:        - /dev/nvme0n1        state: present        vg: vg_nvme        vg_options: ''    item:      key: vg_nvme      value:        pvs: /dev/nvme0n1    msg: Creating volume group 'vg_nvme' failed    rc: 3
2023-07-14T22:35:17.627 INFO:scrape:1 jobs: ['7338126']
2023-07-14T22:35:17.627 INFO:scrape:suites: ['frontend/beast', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/replicated', 's3tests-branch', 'tasks/rgw_user_quota', 'ubuntu_latest}']
2023-07-14T22:35:17.628 INFO:scrape:
2023-07-14T22:35:17.628 INFO:scrape:Failure: smithi148.front.sepia.ceph.com:  _ansible_no_log: false  changed: false  invocation:    module_args:      allow_unauthenticated: false      autoclean: false      autoremove: false      cache_valid_time: 0      deb: null      default_release: null      dpkg_options: force-confdef,force-confold      force: true      force_apt_get: false      install_recommends: null      name:      - ceph      - ceph-common      - libcephfs1      - radosgw      - python-ceph      - python-rados      - python-cephfs      - python-rbd      - librbd1      - librados2      - ceph-fs-common-dbg      - ceph-fs-common      - openmpi-common      only_upgrade: false      package:      - ceph      - ceph-common      - libcephfs1      - radosgw      - python-ceph      - python-rados      - python-cephfs      - python-rbd      - librbd1      - librados2      - ceph-fs-common-dbg      - ceph-fs-common      - openmpi-common      policy_rc_d: null      purge: false      state: absent      update_cache: null      update_cache_retries: 5      update_cache_retry_max_delay: 12      upgrade: null  msg: '''apt-get remove ''librbd1'' ''librados2'''' failed: W: --force-yes is deprecated,    use one of the options starting with --allow instead.    E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 15190    (apt-get)    E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is    another process using it?    '  rc: 100  stderr: 'W: --force-yes is deprecated, use one of the options starting with --allow    instead.    E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 15190    (apt-get)    E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is    another process using it?    '  stderr_lines:  - 'W: --force-yes is deprecated, use one of the options starting with --allow instead.'  - 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 15190    (apt-get)'  - 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is    another process using it?'  stdout: ''  stdout_lines: []
2023-07-14T22:35:17.628 INFO:scrape:1 jobs: ['7338113']
2023-07-14T22:35:17.629 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{single}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{cls', 'validater/lockdep}', 'versioning}']
2023-07-14T22:35:17.629 INFO:scrape:
2023-07-14T22:35:17.629 INFO:scrape:Valgrind: 
2023-07-14T22:35:17.629 INFO:scrape:4 jobs: ['7338097', '7338059', '7338075', '7338076']
2023-07-14T22:35:17.630 INFO:scrape:suites intersection: ['s3tests-branch']
2023-07-14T22:35:17.630 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'http', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'mp_reupload', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/verify/{0-install', 'rgw/website/{clusters/fixed-2', 'rgw_pool_type/ec', 'rgw_pool_type/replicated', 's3tests', 's3tests-branch', 's3tests-java', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{ubuntu_20.04}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/s3tests-website', 'tasks/{cls', 'ubuntu_latest}', 'validater/valgrind}', 'versioning}']
2023-07-14T22:35:17.631 INFO:scrape: