2024-04-03T14:38:19.705 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-03T14:38:19.705 INFO:teuthology.results:subset: '111/120000' 2024-04-03T14:38:19.705 INFO:teuthology.results:seed: '575' 2024-04-04T04:16:13.342 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-04T04:16:13.371 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-04T05:39:20.230 INFO:teuthology.results:Tests finished! gathering results... 2024-04-04T05:39:25.472 INFO:scrape:Found 81 jobs 2024-04-04T05:39:26.685 INFO:scrape:Found 3 distinct failure reasons 2024-04-04T05:39:26.686 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=efebe009020d00364672630ba3da207853319742 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2024-04-04T05:39:26.686 INFO:scrape:2 jobs: ['7638032', '7638100'] 2024-04-04T05:39:26.686 INFO:scrape:suites intersection: ['agent/off', 'mon_election/connectivity', 'task/test_cephadm}'] 2024-04-04T05:39:26.686 INFO:scrape:suites union: ['agent/off', 'mon_election/connectivity', 'orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun', 'orch:cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm}'] 2024-04-04T05:39:26.686 INFO:scrape: 2024-04-04T05:39:26.686 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-04-04T05:39:26.687 INFO:scrape:1 jobs: ['7638072'] 2024-04-04T05:39:26.687 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-04-04T05:39:26.687 INFO:scrape: 2024-04-04T05:39:26.687 INFO:scrape:Failure: '2024-04-04T04:07:35.784180+00:00 smithi178 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog 2024-04-04T05:39:26.687 INFO:scrape:1 jobs: ['7638035'] 2024-04-04T05:39:26.687 INFO:scrape:suites: ['1-bootstrap/16.2.5', '1-start', '2-nfs', '3-upgrade-with-workload', '4-final}', 'orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools'] 2024-04-04T05:39:26.687 INFO:scrape: 2024-04-04T05:39:26.690 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 77 Fail ================================================================= [7638032] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/off mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:12:41 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638032/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95b48bffdda84385baa037abeedecf91 Command failed (workunit test cephadm/test_cephadm.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=efebe009020d00364672630ba3da207853319742 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [7638035] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} ----------------------------------------------------------------- time: 00:45:45 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638035/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638035/ '2024-04-04T04:07:35.784180+00:00 smithi178 kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: ' in syslog [7638072] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:00 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638072/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638072/ reached maximum tries (51) after waiting for 300 seconds [7638100] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:09:42 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638100/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=666d8a2900cc41378e87e1f61a03b03d Command failed (workunit test cephadm/test_cephadm.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=efebe009020d00364672630ba3da207853319742 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' Pass ================================================================= [7638025] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:43:11 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638025/ [7638026] orch:cephadm/no-agent-workunits/{0-distro/rhel_8.6_container_tools_3.0 mon_election/connectivity task/test_cephadm_timeout} time: 00:25:19 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638026/ [7638027] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:21:33 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638027/ [7638028] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:17:47 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638028/ [7638029] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:17:03 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638029/ [7638030] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:46 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638030/ [7638031] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:18:50 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638031/ [7638033] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} time: 00:32:22 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638033/ [7638034] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:18:42 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638034/ [7638036] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} time: 00:47:03 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638036/ [7638037] orch:cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:31:28 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638037/ [7638038] orch:cephadm/no-agent-workunits/{0-distro/rhel_8.6_container_tools_rhel8 mon_election/classic task/test_orch_cli} time: 00:21:14 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638038/ [7638039] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:18:34 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638039/ [7638040] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:21:27 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638040/ [7638041] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/on mon_election/classic task/test_cephadm_repos} time: 00:13:23 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638041/ [7638042] orch:cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:18:56 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638042/ [7638043] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} time: 00:17:29 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638043/ [7638044] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:13:39 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638044/ [7638045] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} time: 00:19:18 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638045/ [7638046] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/off mon_election/connectivity task/test_extra_daemon_features} time: 00:21:43 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638046/ [7638047] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} time: 00:53:31 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638047/ [7638048] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} time: 00:25:47 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638048/ [7638049] orch:cephadm/no-agent-workunits/{0-distro/ubuntu_20.04 mon_election/connectivity task/test_orch_cli_mon} time: 00:35:52 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638049/ [7638050] orch:cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:23:51 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638050/ [7638051] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} time: 00:21:51 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638051/ [7638052] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:15:46 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638052/ [7638053] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:55 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638053/ [7638054] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} time: 00:16:44 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638054/ [7638055] orch:cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_host_drain} time: 00:22:23 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638055/ [7638056] orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} time: 00:58:01 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638056/ [7638057] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs 3-final} time: 00:21:56 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638057/ [7638058] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:51:11 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638058/ [7638059] orch:cephadm/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/nfs} time: 00:34:22 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638059/ [7638060] orch:cephadm/no-agent-workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_adoption} time: 00:17:50 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638060/ [7638061] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr agent/off orchestrator_cli} time: 00:21:18 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638061/ [7638062] orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} time: 00:41:39 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638062/ [7638063] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:20:05 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638063/ [7638064] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs2 3-final} time: 00:16:46 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638064/ [7638065] orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/basic 3-final} time: 00:16:55 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638065/ [7638066] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:15:04 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638066/ [7638067] orch:cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} time: 00:37:33 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638067/ [7638068] orch:cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} time: 00:42:46 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638068/ [7638069] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/connectivity task/test_iscsi_container/{centos_8.stream_container_tools test_iscsi_container}} time: 00:21:58 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638069/ [7638070] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} time: 00:21:20 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638070/ [7638071] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nvmeof 3-final} time: 00:21:24 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638071/ [7638073] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} time: 00:23:41 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638073/ [7638074] orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools_crun agent/on mon_election/classic task/test_rgw_multisite} time: 00:24:03 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638074/ [7638075] orch:cephadm/no-agent-workunits/{0-distro/centos_8.stream_container_tools_crun mon_election/classic task/test_cephadm_timeout} time: 00:20:08 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638075/ [7638076] orch:cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:23:18 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638076/ [7638077] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final} time: 00:18:04 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638077/ [7638078] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} time: 00:49:39 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638078/ [7638079] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} time: 00:31:00 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638079/ [7638080] orch:cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all} time: 00:17:43 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638080/ [7638081] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final} time: 00:17:39 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638081/ [7638082] orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:30 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638082/ [7638083] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_3.0 agent/off mon_election/connectivity task/test_set_mon_crush_locations} time: 00:28:34 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638083/ [7638084] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final} time: 00:18:13 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638084/ [7638085] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:16:31 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638085/ [7638086] orch:cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} time: 00:51:55 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638086/ [7638087] orch:cephadm/no-agent-workunits/{0-distro/rhel_8.6_container_tools_3.0 mon_election/connectivity task/test_orch_cli} time: 00:25:30 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638087/ [7638088] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:21:17 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638088/ [7638089] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/iscsi 3-final} time: 00:21:50 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638089/ [7638090] orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} time: 00:48:12 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638090/ [7638091] orch:cephadm/thrash/{0-distro/rhel_8.6_container_tools_rhel8 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} time: 00:46:05 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638091/ [7638092] orch:cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rotate-keys} time: 00:34:30 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638092/ [7638093] orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/classic task/test_ca_signed_key} time: 00:25:07 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638093/ [7638094] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/jaeger 3-final} time: 00:20:43 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638094/ [7638095] orch:cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:20:16 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638095/ [7638096] orch:cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/mirror 3-final} time: 00:18:03 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638096/ [7638097] orch:cephadm/no-agent-workunits/{0-distro/rhel_8.6_container_tools_rhel8 mon_election/classic task/test_orch_cli_mon} time: 00:33:16 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638097/ [7638098] orch:cephadm/smoke/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:18:44 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638098/ [7638099] orch:cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} time: 00:20:26 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638099/ [7638101] orch:cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli} time: 00:26:44 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638101/ [7638102] orch:cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} time: 00:18:11 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638102/ [7638103] orch:cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 1-start 2-services/rgw 3-final} time: 00:16:58 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638103/ [7638104] orch:cephadm/smoke-small/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:14:49 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638104/ [7638105] orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} time: 01:11:48 info: https://pulpito.ceph.com/yuriw-2024-04-03_14:36:15-orch:cephadm-wip-yuri-testing-2024-04-02-1304-reef-distro-default-smithi/7638105/