2023-05-10T03:48:32.772 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-10T03:48:32.776 INFO:teuthology.results:subset: '699/1024' 2023-05-10T03:48:32.776 INFO:teuthology.results:seed: '5770' 2023-05-10T04:18:20.241 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-10T04:18:20.269 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-10T16:19:17.525 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-05-10T16:19:19.687 INFO:scrape:Found 25 jobs 2023-05-10T16:19:20.919 WARNING:scrape:Misunderstood line: 2023-05-10T16:19:34.446 INFO:scrape:Found 12 distinct failure reasons 2023-05-10T16:19:34.447 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-10T16:19:34.447 INFO:scrape:1 jobs: ['7269338'] 2023-05-10T16:19:34.448 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-10T16:19:34.448 INFO:scrape: 2023-05-10T16:19:34.448 INFO:scrape:Failure: Command failed on smithi194 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4217b7cc-eee9-11ed-9b02-001a4aab830c -- ceph mon dump -f json' 2023-05-10T16:19:34.449 INFO:scrape:1 jobs: ['7269333'] 2023-05-10T16:19:34.449 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-05-10T16:19:34.449 INFO:scrape: 2023-05-10T16:19:34.450 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' 2023-05-10T16:19:34.450 INFO:scrape:3 jobs: ['7269349', '7269327', '7269335'] 2023-05-10T16:19:34.450 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-05-10T16:19:34.451 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-iozone}'] 2023-05-10T16:19:34.451 INFO:scrape: 2023-05-10T16:19:34.451 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-05-10T16:19:34.451 INFO:scrape:1 jobs: ['7269346'] 2023-05-10T16:19:34.452 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-05-10T16:19:34.452 INFO:scrape: 2023-05-10T16:19:34.452 INFO:scrape:Failure: Command failed on smithi159 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-05-10T16:19:34.453 INFO:scrape:4 jobs: ['7269326', '7269348', '7269340', '7269336'] 2023-05-10T16:19:34.453 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-05-10T16:19:34.453 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-05-10T16:19:34.454 INFO:scrape: 2023-05-10T16:19:34.454 INFO:scrape:Dead: b'2023-05-10T15:59:52.794 DEBUG:teuthology.exit:Finished running handlers' 2023-05-10T16:19:34.454 INFO:scrape:1 jobs: ['7269329'] 2023-05-10T16:19:34.455 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-05-10T16:19:34.455 INFO:scrape: 2023-05-10T16:19:34.456 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-05-10T16:19:34.456 INFO:scrape:3 jobs: ['7269350', '7269334', '7269339'] 2023-05-10T16:19:34.456 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-05-10T16:19:34.457 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-05-10T16:19:34.457 INFO:scrape: 2023-05-10T16:19:34.457 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-05-10T16:19:34.458 INFO:scrape:1 jobs: ['7269332'] 2023-05-10T16:19:34.458 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-05-10T16:19:34.458 INFO:scrape: 2023-05-10T16:19:34.458 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-05-10T16:19:34.459 INFO:scrape:1 jobs: ['7269330'] 2023-05-10T16:19:34.459 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-05-10T16:19:34.459 INFO:scrape: 2023-05-10T16:19:34.460 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-05-10T16:19:34.460 INFO:scrape:1 jobs: ['7269342'] 2023-05-10T16:19:34.460 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2023-05-10T16:19:34.461 INFO:scrape: 2023-05-10T16:19:34.461 INFO:scrape:Dead: b'2023-05-10T16:19:24.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-05-10T16:19:34.461 INFO:scrape:1 jobs: ['7269347'] 2023-05-10T16:19:34.461 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-10T16:19:34.462 INFO:scrape: 2023-05-10T16:19:34.462 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-10T16:19:34.462 INFO:scrape:1 jobs: ['7269344'] 2023-05-10T16:19:34.463 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-10T16:19:34.463 INFO:scrape: 2023-05-10T16:19:34.465 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/ failed: 17 dead: 1 running: 1 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7269326] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269326/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269326/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3041103feacc40f9ad87fd245c0f4731 Command failed on smithi159 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7269327] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:21:28 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269327/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1c15f8e1db04f2ab1b98d510a8fbd47 Command failed (workunit test fs/fscrypt.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none ffsb' [7269330] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269330/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269330/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be7a96a1c69f4a65a22b7d67d46c432b Command failed (workunit test fs/quota/quota.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7269332] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:49:45 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269332/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269332/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccda1553352e4af9946823560f006b23 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7269333] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:24 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269333/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269333/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61af63f4771345dda66ca6e8dd8d6341 Command failed on smithi194 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4217b7cc-eee9-11ed-9b02-001a4aab830c -- ceph mon dump -f json' [7269334] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:47:54 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269334/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269334/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7269335] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269335/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56c091069a6147f9825fa16bea241dca Command failed (workunit test fs/fscrypt.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' [7269336] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:57 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269336/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd718bc551cb46ec97c4252cf9439ab3 Command failed on smithi169 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7269338] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:22:34 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269338/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269338/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58218b08fd134de79320da58f3de5c52 Command failed (workunit test fs/test_python.sh) on smithi118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7269339] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:39:11 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269339/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269339/ error during scrub thrashing: rank damage found: {'backtrace'} [7269340] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:18:10 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269340/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a86aba22e5e45fdb9bc9c08da5a2907 Command failed on smithi139 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7269342] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269342/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4df44f1b7bf24822baa42537689df22c Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7269344] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:16 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269344/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269344/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7134f22e2fdd4ce49ed6d4f277207bfd Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7269346] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:00:12 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269346/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269346/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b52ab23f3cdf416d91373bc1a7a55385 saw valgrind issues [7269348] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:18:17 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269348/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269348/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07cb2379888b4d44af15db99eb1c3afa Command failed on smithi107 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7269349] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:11:03 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269349/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e49704205983440ead764dbdf48fe5e2 Command failed (workunit test fs/fscrypt.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38d0174b6b3ad6dc690a927ef6824114d9a01704 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' [7269350] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:53:50 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269350/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269350/ error during scrub thrashing: rank damage found: {'backtrace'} Dead ================================================================= [7269329] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269329/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269329/ hit max job timeout Running ================================================================= [7269347] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269347/ Pass ================================================================= [7269328] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 02:26:16 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269328/ [7269331] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:18:33 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269331/ [7269337] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 09:30:15 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269337/ [7269341] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:44 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269341/ [7269343] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:38:03 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269343/ [7269345] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:47:24 info: http://pulpito.ceph.com/vshankar-2023-05-10_03:47:39-fs-wip-vshankar-testing-20230509.054005-testing-default-smithi/7269345/