2023-03-17T03:01:05.231 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-17T03:01:06.602 INFO:teuthology.results:subset: '111/120000' 2023-03-17T03:01:08.124 INFO:teuthology.results:seed: '5279' 2023-03-17T05:16:13.124 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-17T05:16:15.571 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-17T08:33:28.967 INFO:teuthology.results:Tests finished! gathering results... 2023-03-17T08:33:31.255 INFO:scrape:Found 10 jobs 2023-03-17T08:33:36.618 INFO:scrape:Found 5 distinct failure reasons 2023-03-17T08:33:37.778 INFO:scrape:Failure: Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) 2023-03-17T08:33:37.778 INFO:scrape:2 jobs: ['7211352', '7211358'] 2023-03-17T08:33:37.778 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-03-17T08:33:37.778 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-03-17T08:33:37.778 INFO:scrape: 2023-03-17T08:33:37.778 INFO:scrape:Failure: Command failed on smithi196 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-03-17T08:33:38.457 INFO:scrape:2 jobs: ['7211356', '7211351'] 2023-03-17T08:33:39.130 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-03-17T08:33:39.130 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-03-17T08:33:39.130 INFO:scrape: 2023-03-17T08:33:39.130 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-17T08:33:39.130 INFO:scrape:1 jobs: ['7211353'] 2023-03-17T08:33:39.131 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-17T08:33:39.131 INFO:scrape: 2023-03-17T08:33:39.709 INFO:scrape:Failure: Cannot connect to remote host smithi026 2023-03-17T08:33:39.709 INFO:scrape:2 jobs: ['7211357', '7211350'] 2023-03-17T08:33:39.709 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health'] 2023-03-17T08:33:41.354 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-03-17T08:33:41.354 INFO:scrape: 2023-03-17T08:33:41.354 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi132 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8104bf7ccd3a78474374946451d915b69e7c48eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-03-17T08:33:41.354 INFO:scrape:1 jobs: ['7211349'] 2023-03-17T08:33:41.354 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-03-17T08:33:41.354 INFO:scrape: 2023-03-17T08:33:41.355 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7211349] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 01:10:55 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211349/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a6d49e7c1a4243a4b786915af4ad07da Command failed (workunit test kernel_untar_build.sh) on smithi132 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8104bf7ccd3a78474374946451d915b69e7c48eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7211350] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211350/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211350/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4c9a5376fa8423f8c198b3110e5dbb2 Cannot connect to remote host smithi159 [7211351] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211351/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211351/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3306b77ac3974d3da1da21e559cba9b9 Command failed on smithi017 with status 1: 'set -ex\nsudo dd of=/home/ubuntu/cephtest/valgrind.supp' [7211352] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 02:34:10 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211352/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211352/ sentry: https://sentry.ceph.com/organizations/ceph/?query=93e6f13eba9f424385649d0cf9e25345 Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) [7211353] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 04:00:03 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211353/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211353/ sentry: https://sentry.ceph.com/organizations/ceph/?query=895f73dd00404c2b9ea3071d7e92d831 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7211356] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 01:09:57 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211356/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ba229115eac47b4879728aa790af46c Command failed on smithi196 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7211357] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211357/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211357/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94255d36cc994fbea6e410f3ad6ff74d Cannot connect to remote host smithi026 [7211358] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 02:43:15 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211358/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211358/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a05aacd827614165a1c431cb1da21ec5 Test failure: test_dump_dir (tasks.cephfs.test_misc.TestMisc) Pass ================================================================= [7211354] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 01:40:09 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211354/ [7211355] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:40:23 info: http://pulpito.ceph.com/vshankar-2023-03-17_02:45:20-fs-reef-testing-default-smithi/7211355/