2023-06-20T07:11:54.556 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-20T07:11:54.557 INFO:teuthology.results:subset: '1/14' 2023-06-20T07:11:54.557 INFO:teuthology.results:seed: '2054' 2023-06-20T08:21:12.616 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-20T08:21:12.642 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-20T09:11:16.993 INFO:teuthology.results:Tests finished! gathering results... 2023-06-20T09:11:22.868 INFO:scrape:Found 93 jobs 2023-06-20T09:11:23.431 INFO:scrape:Found 3 distinct failure reasons 2023-06-20T09:11:23.432 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-06-20T09:11:23.432 INFO:scrape:1 jobs: ['7308759'] 2023-06-20T09:11:23.433 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun'] 2023-06-20T09:11:23.433 INFO:scrape: 2023-06-20T09:11:23.433 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6426b360df4b83d5e03a2e3eb352133a987c2ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2023-06-20T09:11:23.434 INFO:scrape:1 jobs: ['7308749'] 2023-06-20T09:11:23.434 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm}'] 2023-06-20T09:11:23.434 INFO:scrape: 2023-06-20T09:11:23.434 INFO:scrape:Failure: Command failed on smithi073 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'set -ex\nFSID=$(/home/ubuntu/cephtest/cephadm shell -- ceph fsid)\n# check extra container and entrypoint args written to mon unit run file\ncat /var/lib/ceph/$FSID/mon.*/unit.run | grep "\\-\\-cpus=2"\ncat /var/lib/ceph/$FSID/mon.*/unit.run | grep "\\-\\-debug_ms 10"\n# check that custom container properly wrote content to file.\n# This requires the custom config, extra container args, and\n# entrypoint args to all be working in order for this to have\n# been written. The container entrypoint was set up with custom_configs,\n# the content and where to write to with the entrypoint args, and the mounting\n# of the /etc/cephadm_testing dir with extra container args\ncat /etc/cephadm_testing/testing.txt | grep "testing_custom_containers"\n\'' 2023-06-20T09:11:23.435 INFO:scrape:1 jobs: ['7308764'] 2023-06-20T09:11:23.435 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'task/test_extra_daemon_features}'] 2023-06-20T09:11:23.435 INFO:scrape: 2023-06-20T09:11:23.439 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 90 Fail ================================================================= [7308749] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308749/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3e4bfe1eb734df595a85cdda5e8c76a Command failed (workunit test cephadm/test_cephadm.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6426b360df4b83d5e03a2e3eb352133a987c2ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7308759] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:30:28 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308759/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27ba2196993943bd8a28e9bacabeb10b reached maximum tries (301) after waiting for 300 seconds [7308764] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_extra_daemon_features} ----------------------------------------------------------------- time: 00:21:35 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308764/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3140bd1640149aaaf3b7e6c3fbabcd0 Command failed on smithi073 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'set -ex\nFSID=$(/home/ubuntu/cephtest/cephadm shell -- ceph fsid)\n# check extra container and entrypoint args written to mon unit run file\ncat /var/lib/ceph/$FSID/mon.*/unit.run | grep "\\-\\-cpus=2"\ncat /var/lib/ceph/$FSID/mon.*/unit.run | grep "\\-\\-debug_ms 10"\n# check that custom container properly wrote content to file.\n# This requires the custom config, extra container args, and\n# entrypoint args to all be working in order for this to have\n# been written. The container entrypoint was set up with custom_configs,\n# the content and where to write to with the entrypoint args, and the mounting\n# of the /etc/cephadm_testing dir with extra container args\ncat /etc/cephadm_testing/testing.txt | grep "testing_custom_containers"\n\'' Pass ================================================================= [7308700] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:44:03 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308700/ [7308701] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:14:54 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308701/ [7308702] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:21:57 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308702/ [7308703] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:16:53 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308703/ [7308704] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:30 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308704/ [7308705] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:41:06 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308705/ [7308706] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli} time: 00:22:53 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308706/ [7308707] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 00:40:31 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308707/ [7308708] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:17:23 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308708/ [7308709] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:13:14 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308709/ [7308710] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:36:42 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308710/ [7308711] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:16:09 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308711/ [7308712] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:01:38 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308712/ [7308713] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_nfs} time: 00:39:46 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308713/ [7308714] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:28:08 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308714/ [7308715] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:38:05 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308715/ [7308716] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} time: 00:34:35 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308716/ [7308717] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:17:38 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308717/ [7308718] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:19 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308718/ [7308719] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:22:27 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308719/ [7308720] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli} time: 00:22:22 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308720/ [7308721] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:20 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308721/ [7308722] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:20:48 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308722/ [7308723] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:38:03 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308723/ [7308724] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:40:10 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308724/ [7308725] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:31:37 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308725/ [7308726] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:15:40 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308726/ [7308727] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:40:49 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308727/ [7308728] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:13:34 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308728/ [7308729] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:16:59 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308729/ [7308730] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:25:15 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308730/ [7308731] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:14:39 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308731/ [7308732] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_rgw_multisite} time: 00:24:31 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308732/ [7308733] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:44:27 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308733/ [7308734] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:28:30 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308734/ [7308735] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:30 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308735/ [7308736] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:45:46 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308736/ [7308737] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:34:47 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308737/ [7308738] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:32:58 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308738/ [7308739] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:39:51 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308739/ [7308740] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:23:25 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308740/ [7308741] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:16:57 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308741/ [7308742] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:16:05 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308742/ [7308743] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:18:55 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308743/ [7308744] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:47:57 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308744/ [7308745] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:42:14 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308745/ [7308746] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_adoption} time: 00:17:15 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308746/ [7308747] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:19:38 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308747/ [7308748] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:22 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308748/ [7308750] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:31:29 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308750/ [7308751] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:15:07 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308751/ [7308752] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools} 2-node-mgr agent/on orchestrator_cli} time: 00:29:41 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308752/ [7308753] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_3.0} 1-start 2-services/rgw 3-final} time: 00:17:06 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308753/ [7308754] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:12:45 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308754/ [7308755] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} time: 01:00:43 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308755/ [7308756] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:34:57 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308756/ [7308757] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:19:05 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308757/ [7308758] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:15:04 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308758/ [7308760] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm_repos} time: 00:09:04 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308760/ [7308761] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:32:19 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308761/ [7308762] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:19:11 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308762/ [7308763] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:15 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308763/ [7308765] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:22:23 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308765/ [7308766] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:43:48 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308766/ [7308767] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:39 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308767/ [7308768] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:39:35 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308768/ [7308769] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:22 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308769/ [7308770] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:25:05 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308770/ [7308771] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:16 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308771/ [7308772] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:23 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308772/ [7308773] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:35:36 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308773/ [7308774] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:16:21 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308774/ [7308775] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:08 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308775/ [7308776] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:13:01 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308776/ [7308777] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:42:00 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308777/ [7308778] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:27:32 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308778/ [7308779] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_nfs} time: 00:38:44 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308779/ [7308780] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:17:29 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308780/ [7308781] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:39:36 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308781/ [7308782] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:14:31 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308782/ [7308783] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:17:51 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308783/ [7308784] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:23:15 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308784/ [7308785] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli} time: 00:18:32 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308785/ [7308786] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:18:06 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308786/ [7308787] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:38:19 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308787/ [7308788] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:44:17 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308788/ [7308789] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:30:19 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308789/ [7308790] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:47 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308790/ [7308791] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:20:34 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308791/ [7308792] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:29:28 info: http://pulpito.ceph.com/adking-2023-06-20_02:10:08-orch:cephadm-wip-adk4-testing-2023-06-19-1637-distro-default-smithi/7308792/