2023-08-24T19:53:00.344 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-24T19:53:00.345 INFO:teuthology.results:subset: '612/20000' 2023-08-24T19:53:00.345 INFO:teuthology.results:seed: '3446' 2023-08-24T20:17:12.544 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-24T20:17:12.568 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-24T21:48:18.789 INFO:teuthology.results:Tests finished! gathering results... 2023-08-24T21:48:22.854 INFO:scrape:Found 40 jobs 2023-08-24T21:48:32.531 INFO:scrape:Found 1 distinct failure reasons 2023-08-24T21:48:32.532 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-08-24T21:48:32.533 INFO:scrape:16 jobs: ['7378857', '7378864', '7378855', '7378872', '7378867', '7378847', '7378862', '7378880', '7378877', '7378859', '7378871', '7378882', '7378868', '7378849', '7378853', '7378850'] 2023-08-24T21:48:32.533 INFO:scrape:suites intersection: [] 2023-08-24T21:48:32.533 INFO:scrape:suites union: ['0-roles', '0-start', '1-start', '1-tasks', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'upgrade/pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'upgrade/pacific-x/parallel/{0-random-distro$/{ubuntu_20.04}', 'upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade/pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade/pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'upgrade/quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0}', 'upgrade/quincy-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade/quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade/quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0', 'upgrade/quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_20.04', 'workload/{ec-rados-default'] 2023-08-24T21:48:32.534 INFO:scrape: 2023-08-24T21:48:32.537 INFO:teuthology.results:Sending results to akupczyk@redhat.com: Test Run: akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 24 Fail ================================================================= [7378847] upgrade/quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:29:44 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378847/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dff45f742f554745becccb86de6ef51a Command failed (workunit test cls/test_cls_rbd.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378849] upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:21:17 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378849/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=040da2a0394c4d048a57d4478f9f365a Command failed (workunit test cls/test_cls_rbd.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378850] upgrade/quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:19:07 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378850/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378850/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffb2c726c09148e6a438e472db7bd059 Command failed (workunit test cls/test_cls_rbd.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378853] upgrade/pacific-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:43:38 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378853/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dccc5fac3e764f47b8be723edccbe5f5 Command failed (workunit test cls/test_cls_rbd.sh) on smithi093 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378855] upgrade/quincy-x/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:40:06 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378855/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eeabe28d87d944dd9e9562c5dc9fea2a Command failed (workunit test cls/test_cls_rbd.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378857] upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:34:36 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378857/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3c71f1b9f9e4fc181935c79a229a004 Command failed (workunit test cls/test_cls_rbd.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378859] upgrade/quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:18:35 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378859/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff020e4928c14d4d82e90d435100ddc9 Command failed (workunit test cls/test_cls_rbd.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378862] upgrade/pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:26:37 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378862/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=274ce3a7aa8e426d90aa074d0310a126 Command failed (workunit test cls/test_cls_rbd.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378864] upgrade/quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 00:27:38 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378864/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b9d2221ad784be2b54489d8ff2f6565 Command failed (workunit test cls/test_cls_rbd.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378867] upgrade/pacific-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:27:24 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378867/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7e8de315c524a07acdefdb8c478012b Command failed (workunit test cls/test_cls_rbd.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378868] upgrade/quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_rhel8 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:20:49 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378868/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af4a1524648c4862bf6bb5d359e159b4 Command failed (workunit test cls/test_cls_rbd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378871] upgrade/pacific-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:22:04 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378871/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f5ee06fcd1847dcaf7490cb402aa9a5 Command failed (workunit test cls/test_cls_rbd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378872] upgrade/quincy-x/stress-split/{0-distro/ubuntu_20.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} ----------------------------------------------------------------- time: 01:26:35 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378872/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b84f7947552a442283c9b2e101164788 Command failed (workunit test cls/test_cls_rbd.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378877] upgrade/pacific-x/parallel/{0-random-distro$/{ubuntu_20.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:44:08 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378877/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f202e9a5aae64424b90de5f284dec30f Command failed (workunit test cls/test_cls_rbd.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=pacific TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378880] upgrade/quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:44:15 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378880/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f168b2fcaa4248849cfd73dd40f91329 Command failed (workunit test cls/test_cls_rbd.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7378882] upgrade/pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} ----------------------------------------------------------------- time: 01:12:28 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378882/ log: http://qa-proxy.ceph.com/teuthology/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378882/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89102838c35346409a1edcd4d9b325b4 Command failed (workunit test cls/test_cls_rbd.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' Pass ================================================================= [7378845] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:20:22 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378845/ [7378846] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:21 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378846/ [7378848] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:19:02 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378848/ [7378851] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:14:37 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378851/ [7378852] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:09 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378852/ [7378854] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:14:44 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378854/ [7378856] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:39 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378856/ [7378858] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:25:42 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378858/ [7378860] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:27 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378860/ [7378861] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:53:49 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378861/ [7378863] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/fsstress}} time: 00:19:33 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378863/ [7378865] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:02 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378865/ [7378866] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:46 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378866/ [7378869] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/iozone}} time: 00:18:10 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378869/ [7378870] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/kernel_untar_build}} time: 01:17:49 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378870/ [7378873] upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks} time: 00:28:15 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378873/ [7378874] upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:12 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378874/ [7378875] upgrade/cephfs/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:15:07 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378875/ [7378876] upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_20.04} time: 00:09:02 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378876/ [7378878] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/new_ops/client-sanity}} time: 00:13:50 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378878/ [7378879] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-pacific 1-client-upgrade 2-workload/stress_tests/blogbench}} time: 00:24:55 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378879/ [7378881] upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:16 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378881/ [7378883] upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:52 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378883/ [7378884] upgrade/cephfs/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/0-nautilus 1-client-upgrade 2-workload/stress_tests/dbench}} time: 00:55:33 info: http://pulpito.ceph.com/akupczyk-2023-08-24_18:39:36-upgrade-wip-aclamk-bs-esb-5-enabled-distro-default-smithi/7378884/