2023-05-20T12:05:31.103 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-20T12:05:31.104 INFO:teuthology.results:subset: '111/12000'
2023-05-20T12:05:31.104 INFO:teuthology.results:seed: '9016'
2023-05-20T13:14:25.272 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e
2023-05-20T13:14:25.300 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-20T18:37:52.136 INFO:teuthology.results:Tests finished! gathering results...
2023-05-20T18:37:57.561 INFO:scrape:Found 41 jobs
2023-05-20T18:38:00.748 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279840/teuthology.log matches
2023-05-20T18:38:00.750 WARNING:scrape:Misunderstood line: 
2023-05-20T18:38:04.942 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279888/teuthology.log matches
2023-05-20T18:38:04.943 WARNING:scrape:Misunderstood line: 
2023-05-20T18:38:04.944 INFO:scrape:Found 3 distinct failure reasons
2023-05-20T18:38:04.945 INFO:scrape:Failure: rgw multisite test failures
2023-05-20T18:38:04.945 INFO:scrape:2 jobs: ['7279858', '7279819']
2023-05-20T18:38:04.946 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}']
2023-05-20T18:38:04.946 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}']
2023-05-20T18:38:04.947 INFO:scrape:
2023-05-20T18:38:04.948 INFO:scrape:Failure: Command failed on smithi119 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'
2023-05-20T18:38:04.948 INFO:scrape:1 jobs: ['7279868']
2023-05-20T18:38:04.949 INFO:scrape:suites: ['frontend/beast', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}']
2023-05-20T18:38:04.949 INFO:scrape:
2023-05-20T18:38:04.950 INFO:scrape:Valgrind: 
2023-05-20T18:38:04.951 INFO:scrape:4 jobs: ['7279840', '7279872', '7279876', '7279888']
2023-05-20T18:38:04.951 INFO:scrape:suites intersection: []
2023-05-20T18:38:04.952 INFO:scrape:suites union: ['cluster', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/tools/{centos_latest', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'tasks}', 'validater/lockdep}', 'validater/valgrind}']
2023-05-20T18:38:04.953 INFO:scrape:
2023-05-20T18:38:04.956 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  34


Fail
=================================================================
[7279819]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi}
-----------------------------------------------------------------
time:   02:09:11
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279819/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279819/
sentry: https://sentry.ceph.com/organizations/ceph/?query=412e47e5d7c04195b9976972c8955c90

    rgw multisite test failures

[7279840]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:09:31
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279840/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279840/

    saw valgrind issues

[7279858]  rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_20.04} tasks/test_multi}
-----------------------------------------------------------------
time:   03:25:10
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279858/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279858/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cc03e63622646469b7615b28c89c4ca

    rgw multisite test failures

[7279868]  rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:59
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279868/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279868/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b06a8759964f451b82b1a34e6e88a913

    Command failed on smithi119 with status 1: 'cd
    /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-
    venv/bin/activate && tox -e venv --notest'

[7279872]  rgw/tools/{centos_latest cluster tasks}
-----------------------------------------------------------------
time:   05:37:56
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279872/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279872/

    Exiting scrub checking -- not all pgs scrubbed.

[7279876]  rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
-----------------------------------------------------------------
time:   00:49:20
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279876/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279876/

    "1684588637.199595 mon.a (mon.0) 334 : cluster [WRN] Health check failed: 1
    pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in
    cluster log

[7279888]  rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind}
-----------------------------------------------------------------
time:   01:21:11
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279888/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279888/

    "1684589499.7913404 mon.a (mon.0) 417 : cluster [WRN] Health check failed:
    1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in
    cluster log



Pass
=================================================================
[7279817] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop}
time:   00:20:52
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279817/

[7279821] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests}
time:   00:32:28
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279821/

[7279823] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:25:29
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279823/

[7279825] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep}
time:   00:41:32
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279825/

[7279828] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:59
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279828/

[7279830] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279830/

[7279832] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}}
time:   00:22:25
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279832/

[7279834] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:43
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279834/

[7279836] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota}
time:   00:30:33
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279836/

[7279838] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279838/

[7279842] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:24:47
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279842/

[7279844] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest}
time:   00:12:21
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279844/

[7279846] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests}
time:   00:46:34
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279846/

[7279848] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:30:12
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279848/

[7279850] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:25:33
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279850/

[7279852] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop}
time:   00:21:52
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279852/

[7279854] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests}
time:   00:26:05
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279854/

[7279856] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest}
time:   00:13:45
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279856/

[7279860] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}}
time:   00:53:08
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279860/

[7279862] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest}
time:   00:26:37
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279862/

[7279864] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}}
time:   00:18:39
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279864/

[7279866] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_8} tasks/{0-install first ststests}}
time:   00:31:27
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279866/

[7279870] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota}
time:   00:47:18
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279870/

[7279874] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279874/

[7279878] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest}
time:   00:32:57
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279878/

[7279880] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest}
time:   00:12:53
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279880/

[7279882] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:23:36
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279882/

[7279884] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload}
time:   00:27:54
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279884/

[7279886] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides}
time:   00:13:35
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279886/

[7279890] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest}
time:   00:11:31
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279890/

[7279892] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}}
time:   00:18:15
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279892/

[7279894] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}}
time:   00:15:16
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279894/

[7279896] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest}
time:   00:20:35
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279896/

[7279898] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}}
time:   00:15:19
info:   http://pulpito.ceph.com/yuriw-2023-05-19_20:44:42-rgw-reef-distro-default-smithi/7279898/