2023-07-13T00:34:57.221 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-13T00:34:57.222 INFO:teuthology.results:subset: '1/8' 2023-07-13T00:34:57.223 INFO:teuthology.results:seed: '4977' 2023-07-13T00:45:03.030 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-13T00:45:03.040 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-13T01:52:08.625 INFO:teuthology.results:Tests finished! gathering results... 2023-07-13T01:52:10.234 INFO:scrape:Found 39 jobs 2023-07-13T01:52:10.988 INFO:scrape:Found 2 distinct failure reasons 2023-07-13T01:52:10.989 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-07-13T01:52:10.989 INFO:scrape:2 jobs: ['7335559', '7335522'] 2023-07-13T01:52:10.990 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}'] 2023-07-13T01:52:10.990 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-07-13T01:52:10.990 INFO:scrape: 2023-07-13T01:52:10.991 INFO:scrape:Failure: Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) 2023-07-13T01:52:10.991 INFO:scrape:1 jobs: ['7335524'] 2023-07-13T01:52:10.991 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8', 'task/test_orch_cli}'] 2023-07-13T01:52:10.991 INFO:scrape: 2023-07-13T01:52:10.994 INFO:teuthology.results:Sending results to gabrioux@redhat.com: Test Run: gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 36 Fail ================================================================= [7335522] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:39:06 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335522/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaf09ef1868d4d698c296b220665ae23 reached maximum tries (301) after waiting for 300 seconds [7335524] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} ----------------------------------------------------------------- time: 00:21:09 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335524/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2069962ed844ec788383e82db2cfa71 Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI) [7335559] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:32:51 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335559/ log: http://qa-proxy.ceph.com/teuthology/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=429bdcc3f45d4d38aa499194baf032e6 reached maximum tries (301) after waiting for 300 seconds Pass ================================================================= [7335523] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:47:41 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335523/ [7335525] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:50 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335525/ [7335526] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli_mon} time: 00:36:50 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335526/ [7335527] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:19:23 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335527/ [7335528] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_set_mon_crush_locations} time: 00:43:29 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335528/ [7335529] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:02:36 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335529/ [7335530] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:51:25 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335530/ [7335531] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/connectivity task/test_adoption} time: 00:19:58 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335531/ [7335532] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_cephadm} time: 00:23:10 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335532/ [7335533] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:18:27 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335533/ [7335534] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:44:21 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335534/ [7335535] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:55:58 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335535/ [7335536] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:54:04 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335536/ [7335537] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:31:42 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335537/ [7335538] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:43:13 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335538/ [7335539] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_nfs} time: 00:48:36 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335539/ [7335540] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:39:18 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335540/ [7335541] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_orch_cli} time: 00:23:28 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335541/ [7335542] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:43:13 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335542/ [7335543] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:35:12 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335543/ [7335544] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:23:25 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335544/ [7335545] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:50:01 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335545/ [7335546] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} time: 00:39:24 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335546/ [7335547] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:20:15 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335547/ [7335548] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:50:48 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335548/ [7335549] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_rgw_multisite} time: 00:19:02 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335549/ [7335550] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:38:04 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335550/ [7335551] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:17:04 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335551/ [7335552] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:46:23 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335552/ [7335553] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:54:22 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335553/ [7335554] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/classic task/test_adoption} time: 00:23:35 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335554/ [7335555] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:53:46 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335555/ [7335556] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} time: 00:40:26 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335556/ [7335557] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:46:32 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335557/ [7335558] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:22:14 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335558/ [7335560] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli} time: 00:28:10 info: http://pulpito.ceph.com/gabrioux-2023-07-13_00:34:27-orch:cephadm-wip-guits-testing-4-2023-07-12-0746-distro-default-smithi/7335560/