2023-10-03T18:45:24.040 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T18:45:24.040 INFO:teuthology.results:seed: '3306' 2023-10-03T19:21:34.965 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T19:21:34.992 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-03T20:29:40.145 INFO:teuthology.results:Tests finished! gathering results... 2023-10-03T20:29:41.853 INFO:scrape:Found 24 jobs 2023-10-03T20:29:49.101 INFO:scrape:Found 3 distinct failure reasons 2023-10-03T20:29:49.101 INFO:scrape:Failure: Command failed on smithi148 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-10-03T20:29:49.102 INFO:scrape:16 jobs: ['7409799', '7409791', '7409800', '7409806', '7409796', '7409805', '7409794', '7409787', '7409802', '7409797', '7409790', '7409808', '7409803', '7409788', '7409809', '7409793'] 2023-10-03T20:29:49.102 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs:multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-10-03T20:29:49.102 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs:multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-10-03T20:29:49.102 INFO:scrape: 2023-10-03T20:29:49.102 INFO:scrape:Failure: Test failure: test_dump_inmemory_log_on_client_eviction (tasks.cephfs.test_misc.TestMisc) 2023-10-03T20:29:49.102 INFO:scrape:1 jobs: ['7409792'] 2023-10-03T20:29:49.103 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs:multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-10-03T20:29:49.103 INFO:scrape: 2023-10-03T20:29:49.103 INFO:scrape:Failure: Test failure: test_dump_inmemory_log_on_missed_beacon_ack_from_monitors (tasks.cephfs.test_misc.TestMisc) 2023-10-03T20:29:49.103 INFO:scrape:1 jobs: ['7409789'] 2023-10-03T20:29:49.103 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs:multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-10-03T20:29:49.103 INFO:scrape: 2023-10-03T20:29:49.104 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7409787] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:17:37 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409787/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b90d292773404fafa95ff7c16a3e30b4 Command failed on smithi134 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409788] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:52 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409788/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=809b2bfd11944211ba6d47df08e7a308 Command failed on smithi144 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409789] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:24:29 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409789/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409789/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53f1b41b799a454e99ddde1d83fec118 Test failure: test_dump_inmemory_log_on_missed_beacon_ack_from_monitors (tasks.cephfs.test_misc.TestMisc) [7409790] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:18:28 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409790/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409790/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c977e2aafc744235b67ccf1fde36cc55 Command failed on smithi112 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409791] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:18:39 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409791/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90caddade1f84a95ad705cf35d924d8e Command failed on smithi129 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409792] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:22:10 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409792/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8028949eec294109855b96721f3cef6e Test failure: test_dump_inmemory_log_on_client_eviction (tasks.cephfs.test_misc.TestMisc) [7409793] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:17:48 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409793/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65fa9e6ff5034df2b33be0aab2b30d77 Command failed on smithi174 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409794] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409794/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b8650df00c024b25a309a6667239c3ad Command failed on smithi142 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409796] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:39 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409796/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0dfcae47c9d74a5888080995a7c67230 Command failed on smithi191 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409797] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:19:32 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409797/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=074430648ff74f0689bde49cffeb2ccf Command failed on smithi098 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409799] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:19:05 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409799/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae4501c1bbb147dc883e3dfed9ddad28 Command failed on smithi148 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409800] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:31 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409800/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbbbd504d12d4c5b9277a8c33a9c23ca Command failed on smithi125 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409802] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:00 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409802/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409802/ sentry: https://sentry.ceph.com/organizations/ceph/?query=376391d9cf12410d9ba5eab4aeecf08c Command failed on smithi083 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409803] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:29 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409803/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=207f2a347b1547b6b2686d5b1b153a44 Command failed on smithi195 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409805] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:17:28 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409805/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88462c0109b546b6bdf4d6ea3a9a00d4 Command failed on smithi071 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409806] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:50 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409806/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b9e6f32adf545d6ae3d1dcce0e1160b Command failed on smithi112 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409808] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:35 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409808/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=434613143169400faf30103255c99cbd Command failed on smithi094 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7409809] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:43 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409809/ log: http://qa-proxy.ceph.com/teuthology/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76d7b822d1a84663934b1c6ae16220a1 Command failed on smithi178 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' Pass ================================================================= [7409786] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:16 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409786/ [7409795] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:02:20 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409795/ [7409798] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:02:20 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409798/ [7409801] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:33 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409801/ [7409804] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:06 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409804/ [7409807] fs:multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:33 info: http://pulpito.ceph.com/choffman-2023-10-03_18:32:29-fs:multiclient-wip-choffman-62793-distro-default-smithi/7409807/