2023-08-28T17:56:30.936 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-28T17:56:30.936 INFO:teuthology.results:subset: '1/14' 2023-08-28T17:56:30.936 INFO:teuthology.results:seed: '5920' 2023-08-28T18:41:09.408 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-28T18:41:09.418 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-28T20:35:18.194 INFO:teuthology.results:Tests finished! gathering results... 2023-08-28T20:35:23.969 INFO:scrape:Found 95 jobs 2023-08-28T20:35:23.970 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382348/teuthology.log 2023-08-28T20:35:26.510 INFO:scrape:Found 11 distinct failure reasons 2023-08-28T20:35:26.510 INFO:scrape:Dead: None 2023-08-28T20:35:26.510 INFO:scrape:1 jobs: ['7382348'] 2023-08-28T20:35:26.510 INFO:scrape: 2023-08-28T20:35:26.511 INFO:scrape:Failure: Command failed on smithi031 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.31:/foobucket /mnt/foo'" 2023-08-28T20:35:26.511 INFO:scrape:3 jobs: ['7382286', '7382354', '7382359'] 2023-08-28T20:35:26.511 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '3-final}'] 2023-08-28T20:35:26.511 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-bucket', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8', 'orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04'] 2023-08-28T20:35:26.511 INFO:scrape: 2023-08-28T20:35:26.511 INFO:scrape:Failure: reached maximum tries (301) after waiting for 300 seconds 2023-08-28T20:35:26.511 INFO:scrape:1 jobs: ['7382342'] 2023-08-28T20:35:26.511 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/jaeger', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools'] 2023-08-28T20:35:26.512 INFO:scrape: 2023-08-28T20:35:26.512 INFO:scrape:Failure: Command failed on smithi033 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7ad454e2787b18b20fbc6be4ced6d1b76d089c0a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8121935c-45d3-11ee-9b8b-d70336b2926f -- ceph osd stat -f json' 2023-08-28T20:35:26.512 INFO:scrape:1 jobs: ['7382372'] 2023-08-28T20:35:26.512 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-keepalive-only', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-08-28T20:35:26.512 INFO:scrape: 2023-08-28T20:35:26.512 INFO:scrape:Failure: Command failed on smithi115 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4bfec7ee-45d3-11ee-9b8b-d70336b2926f -- bash -c 'ceph orch upgrade ls'" 2023-08-28T20:35:26.512 INFO:scrape:4 jobs: ['7382365', '7382290', '7382322', '7382302'] 2023-08-28T20:35:26.512 INFO:scrape:suites intersection: [] 2023-08-28T20:35:26.512 INFO:scrape:suites union: ['1-bootstrap/16.2.0', '1-start', '2-nfs', '2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-upgrade-with-workload', '3-upgrade/simple', '3-upgrade/staggered', '4-final}', '4-wait', '5-upgrade-ls', 'agent/off', 'agent/on', 'mon_election/classic}', 'mon_election/connectivity}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-08-28T20:35:26.513 INFO:scrape: 2023-08-28T20:35:26.513 INFO:scrape:Failure: '2023-08-28T19:08:51.611735+00:00 smithi070 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog 2023-08-28T20:35:26.513 INFO:scrape:1 jobs: ['7382355'] 2023-08-28T20:35:26.513 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2023-08-28T20:35:26.513 INFO:scrape: 2023-08-28T20:35:26.513 INFO:scrape:Failure: machine smithi002.front.sepia.ceph.com is locked by scheduled_zack@teuthology, not scheduled_adking@teuthology 2023-08-28T20:35:26.513 INFO:scrape:1 jobs: ['7382351'] 2023-08-28T20:35:26.513 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-haproxy-proto', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0'] 2023-08-28T20:35:26.514 INFO:scrape: 2023-08-28T20:35:26.514 INFO:scrape:Failure: ['Failed to connect to the host via ssh: ssh: connect to host smithi002.front.sepia.ceph.com port 22: No route to host'] 2023-08-28T20:35:26.514 INFO:scrape:1 jobs: ['7382341'] 2023-08-28T20:35:26.514 INFO:scrape:suites: ['1-start', '2-thrash', '3-tasks/radosbench', 'fixed-2', 'msgr/async-v2only', 'orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8', 'root}'] 2023-08-28T20:35:26.514 INFO:scrape: 2023-08-28T20:35:26.514 INFO:scrape:Failure: 2023-08-28T20:35:26.514 INFO:scrape:1 jobs: ['7382343'] 2023-08-28T20:35:26.514 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-upgrade/simple', '4-wait', '5-upgrade-ls', 'agent/off', 'mon_election/connectivity}', 'orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04'] 2023-08-28T20:35:26.514 INFO:scrape: 2023-08-28T20:35:26.515 INFO:scrape:Failure: ["dd: failed to open '/home/ubuntu/.ansible/tmp/ansible-tmp-1693245853.2524476-26393-9428471280628/AnsiballZ_user.py': No such file or directory", 'failed to transfer file to /home/teuthworker/.ansible/tmp/ansible-local-23711q6nmx9w7/tmph9k0x4lu /home/ubuntu/.ansible/tmp/ansible-tmp-1693245853.2524476-26393-9428471280628/AnsiballZ_user.py:'] 2023-08-28T20:35:26.515 INFO:scrape:1 jobs: ['7382295'] 2023-08-28T20:35:26.515 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-services/nfs-ingress-rgw-user', '3-final}', 'orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8'] 2023-08-28T20:35:26.515 INFO:scrape: 2023-08-28T20:35:26.515 INFO:scrape:Failure: Stale jobs detected, aborting. 2023-08-28T20:35:26.515 INFO:scrape:1 jobs: ['7382296'] 2023-08-28T20:35:26.515 INFO:scrape:suites: ['fixed-2', 'mode/root', 'mon_election/connectivity', 'msgr/async-v2only', 'orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8', 'start', 'tasks/rados_python}'] 2023-08-28T20:35:26.515 INFO:scrape: 2023-08-28T20:35:26.518 INFO:teuthology.results:Sending results to adking@redhat.com: Test Run: adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/ failed: 14 dead: 2 running: 0 waiting: 0 queued: 0 passed: 79 Fail ================================================================= [7382286] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:25:29 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382286/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382286/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff7a8200fc7a4528aa432f717f7428ac Command failed on smithi031 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.31:/foobucket /mnt/foo'" [7382290] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:20:54 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382290/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=074174aa34bd40b9a3df81b6633bd969 Command failed on smithi037 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55c633ce-45cf-11ee-9b8b-d70336b2926f -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done'" [7382296] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382296/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b22026d724fd4ce88c77c855425b8555 Stale jobs detected, aborting. [7382302] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} ----------------------------------------------------------------- time: 00:39:12 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382302/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382302/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee4048dc8b0642249a3c0d38d98a32ab Command failed on smithi155 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 803d7f7c-45cf-11ee-9b8b-d70336b2926f -- bash -c 'ceph orch upgrade ls'" [7382322] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} ----------------------------------------------------------------- time: 00:46:33 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382322/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382322/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccfcab69ab344f4eb5f28fad74e527d5 Command failed on smithi084 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cd92160c-45cf-11ee-9b8b-d70336b2926f -- bash -c 'ceph orch upgrade ls'" [7382341] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} ----------------------------------------------------------------- time: 00:11:43 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382341/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382341/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04ebed09ca824605baf4adb6c3d15cfa ['Failed to connect to the host via ssh: ssh: connect to host smithi002.front.sepia.ceph.com port 22: No route to host'] [7382342] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/jaeger 3-final} ----------------------------------------------------------------- time: 00:29:02 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382342/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fea148e103f4a778e161b88db6b2ab1 reached maximum tries (301) after waiting for 300 seconds [7382343] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382343/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382343/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98e1369e545a44f8852f104d4fb76fd3 [7382351] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382351/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=825644cab8f14e64aa516aaf2cb9e748 machine smithi002.front.sepia.ceph.com is locked by scheduled_zack@teuthology, not scheduled_adking@teuthology [7382354] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} ----------------------------------------------------------------- time: 00:20:24 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382354/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=807334e7eda14e41811fd36d120e9e02 Command failed on smithi081 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.81:/foobucket /mnt/foo'" [7382355] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:39:55 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382355/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382355/ '2023-08-28T19:08:51.611735+00:00 smithi070 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog [7382359] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:23:48 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382359/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382359/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e892869062524649b30e6bfb9b395490 Command failed on smithi107 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.107:/foouser /mnt/foo'" [7382365] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} ----------------------------------------------------------------- time: 00:26:48 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382365/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382365/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11726e75be4b43a9998e4b7ac85cebe2 Command failed on smithi115 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4bfec7ee-45d3-11ee-9b8b-d70336b2926f -- bash -c 'ceph orch upgrade ls'" [7382372] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382372/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29c5d2dcfd114f688765ab9d7c4da102 Command failed on smithi033 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:7ad454e2787b18b20fbc6be4ced6d1b76d089c0a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8121935c-45d3-11ee-9b8b-d70336b2926f -- ceph osd stat -f json' Dead ================================================================= [7382295] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} ----------------------------------------------------------------- time: 00:09:56 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382295/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382295/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f9940b56e7f41ff982866f981d88e65 ["dd: failed to open '/home/ubuntu/.ansible/tmp/ansible- tmp-1693245853.2524476-26393-9428471280628/AnsiballZ_user.py': No such file or directory", 'failed to transfer file to /home/teuthworker/.ansible/tmp/ansible-local-23711q6nmx9w7/tmph9k0x4lu /home/ubuntu/.ansible/tmp/ansible- tmp-1693245853.2524476-26393-9428471280628/AnsiballZ_user.py:'] [7382348] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/mirror 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382348/ log: http://qa-proxy.ceph.com/teuthology/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382348/ Error reimaging machines: Expected smithi002's OS to be centos 8 but found rhel 8.6 Pass ================================================================= [7382287] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:17:41 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382287/ [7382288] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:21:48 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382288/ [7382289] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:17:47 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382289/ [7382291] orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli} time: 00:20:10 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382291/ [7382292] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_iscsi} time: 00:43:25 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382292/ [7382293] orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/basic 3-final} time: 00:20:22 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382293/ [7382294] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:18:14 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382294/ [7382297] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:26:31 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382297/ [7382298] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:06:50 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382298/ [7382299] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_nfs} time: 00:39:35 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382299/ [7382300] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} time: 00:42:15 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382300/ [7382301] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:20:10 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382301/ [7382303] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:21:13 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382303/ [7382304] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:20:49 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382304/ [7382305] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli} time: 00:17:45 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382305/ [7382306] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:18:41 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382306/ [7382307] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:17 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382307/ [7382308] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} time: 00:35:07 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382308/ [7382309] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:39:57 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382309/ [7382310] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:23:03 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382310/ [7382311] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:29:51 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382311/ [7382312] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:21:41 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382312/ [7382313] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:26:28 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382313/ [7382314] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:18:18 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382314/ [7382315] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:22:47 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382315/ [7382316] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:29:14 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382316/ [7382317] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_rgw_multisite} time: 00:24:11 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382317/ [7382318] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:18:39 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382318/ [7382319] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} time: 00:40:36 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382319/ [7382320] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:21 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382320/ [7382321] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:17:34 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382321/ [7382323] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:37:10 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382323/ [7382324] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_set_mon_crush_locations} time: 00:29:21 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382324/ [7382325] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:41:30 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382325/ [7382326] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:17:39 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382326/ [7382327] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} time: 00:20:32 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382327/ [7382328] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:17:33 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382328/ [7382329] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:20:03 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382329/ [7382330] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} time: 00:48:52 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382330/ [7382331] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:47:06 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382331/ [7382332] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/classic task/test_adoption} time: 00:21:18 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382332/ [7382333] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:21:50 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382333/ [7382334] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:32 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382334/ [7382335] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/connectivity task/test_ca_signed_key} time: 00:18:20 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382335/ [7382336] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:33:55 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382336/ [7382337] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:20:28 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382337/ [7382338] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_3.0} 2-node-mgr agent/on orchestrator_cli} time: 00:25:40 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382338/ [7382339] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/rgw 3-final} time: 00:15:52 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382339/ [7382340] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:14:31 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382340/ [7382344] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:09 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382344/ [7382345] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:16:03 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382345/ [7382346] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_cephadm} time: 00:19:16 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382346/ [7382347] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} time: 00:26:42 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382347/ [7382349] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:50 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382349/ [7382350] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:09:27 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382350/ [7382352] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} time: 00:47:01 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382352/ [7382353] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:37:15 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382353/ [7382356] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:15:26 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382356/ [7382357] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:19:45 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382357/ [7382358] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/classic task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:18:01 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382358/ [7382360] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:08 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382360/ [7382361] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:45 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382361/ [7382362] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:18:02 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382362/ [7382363] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:14:32 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382363/ [7382364] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} time: 00:38:15 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382364/ [7382366] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_nfs} time: 00:35:16 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382366/ [7382367] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} time: 00:29:39 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382367/ [7382368] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:16:56 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382368/ [7382369] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:15:26 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382369/ [7382370] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:18:37 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382370/ [7382371] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/classic task/test_orch_cli} time: 00:19:00 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382371/ [7382373] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:37:52 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382373/ [7382374] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:43:57 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382374/ [7382375] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:18:17 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382375/ [7382376] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/on mon_election/connectivity task/test_orch_cli_mon} time: 00:24:36 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382376/ [7382377] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:44 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382377/ [7382378] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:18:40 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382378/ [7382379] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:24:43 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382379/ [7382380] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:19:23 info: http://pulpito.ceph.com/adking-2023-08-28_16:10:32-orch:cephadm-wip-adk4-testing-2023-08-23-1835-reef-distro-default-smithi/7382380/