2024-02-29T15:15:59.725 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-29T15:15:59.725 INFO:teuthology.results:subset: '788/1024' 2024-02-29T15:15:59.725 INFO:teuthology.results:seed: '1879' 2024-02-29T16:08:01.548 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-29T16:08:01.567 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-01T03:52:53.921 INFO:teuthology.results:Tests finished! gathering results... 2024-03-01T03:52:59.502 INFO:scrape:Found 61 jobs 2024-03-01T03:53:00.744 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.744 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.744 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.745 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.818 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.818 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.818 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.818 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.895 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.895 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.895 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.895 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.969 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.969 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.969 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:00.969 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.043 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.043 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.044 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.044 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.116 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.116 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.117 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.117 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.189 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.190 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.190 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:01.190 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:13.136 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576695/teuthology.log 2024-03-01T03:53:14.392 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.392 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.392 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.393 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.485 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.485 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.485 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.486 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.577 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.577 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.577 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.577 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.668 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.668 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.668 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:14.668 WARNING:scrape:Saw end of BT but not start 2024-03-01T03:53:19.507 INFO:scrape:Found 12 distinct failure reasons 2024-03-01T03:53:19.508 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-03-01T03:53:19.508 INFO:scrape:3 jobs: ['7576712', '7576681', '7576706'] 2024-03-01T03:53:19.508 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse'] 2024-03-01T03:53:19.508 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'ranks/1', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'tasks/mirror}}', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-03-01T03:53:19.508 INFO:scrape: 2024-03-01T03:53:19.508 INFO:scrape:Failure: Command failed on smithi116 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 04eef8aa-d71f-11ee-95c3-87774f69a715 -e sha1=6085b91fefeb74bf4310344a87d81d3217ef7893 -- bash -c 'ceph orch ps'" 2024-03-01T03:53:19.508 INFO:scrape:2 jobs: ['7576718', '7576705'] 2024-03-01T03:53:19.509 INFO:scrape:suites intersection: ['1-volume/{0-create', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-01T03:53:19.509 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '2-client/kclient', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{reef}'] 2024-03-01T03:53:19.509 INFO:scrape: 2024-03-01T03:53:19.509 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-03-01T03:53:19.509 INFO:scrape:1 jobs: ['7576679'] 2024-03-01T03:53:19.509 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-01T03:53:19.509 INFO:scrape: 2024-03-01T03:53:19.509 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi184.front.sepia.ceph.com: ['type=AVC msg=audit(1709223094.001:210): avc: denied { checkpoint_restore } for pid=1229 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-01T03:53:19.509 INFO:scrape:13 jobs: ['7576708', '7576678', '7576709', '7576710', '7576661', '7576680', '7576690', '7576685', '7576716', '7576671', '7576675', '7576672', '7576694'] 2024-03-01T03:53:19.509 INFO:scrape:suites intersection: [] 2024-03-01T03:53:19.510 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '1-mount', '1-ranks/2', '1-rename', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '2-workunit/iozone}}', '2-workunit/suites/fsstress}}', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/blogbench}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'centos_latest', 'clusters/1-node-1-mds-1-osd', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/small-cluster', 'conf/{client', 'distro/single-container-host', 'distro/single-container-host}', 'distro/{centos_latest}', 'export-check', 'fail_fs/yes', 'frag', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/cephadm/multivolume/{0-start', 'fs/cephadm/renamevolume/{0-start', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/verify/{begin/{0-install', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'ranks/5', 'ranks/multi/{balancer/automatic', 'replication/always}', 'roles', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvol_versions/create_subvol_version_v1', 'subvol_versions/create_subvol_version_v2', 'syntax}', 'tasks/cap-flush}', 'tasks/dbench', 'tasks/journal-repair}', 'tasks/mgr-osd-full}', 'tasks/trim-i24137}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'validater/lockdep}', 'wsync/yes}'] 2024-03-01T03:53:19.510 INFO:scrape: 2024-03-01T03:53:19.510 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi114 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-01T03:53:19.510 INFO:scrape:2 jobs: ['7576697', '7576673'] 2024-03-01T03:53:19.510 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay'] 2024-03-01T03:53:19.510 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-03-01T03:53:19.510 INFO:scrape: 2024-03-01T03:53:19.510 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-03-01T03:53:19.510 INFO:scrape:1 jobs: ['7576704'] 2024-03-01T03:53:19.510 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-03-01T03:53:19.511 INFO:scrape: 2024-03-01T03:53:19.511 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-01T03:53:19.511 INFO:scrape:1 jobs: ['7576700'] 2024-03-01T03:53:19.511 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2024-03-01T03:53:19.511 INFO:scrape: 2024-03-01T03:53:19.511 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-01T03:53:19.511 INFO:scrape:1 jobs: ['7576687'] 2024-03-01T03:53:19.511 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2024-03-01T03:53:19.511 INFO:scrape: 2024-03-01T03:53:19.511 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-01T03:53:19.511 INFO:scrape:4 jobs: ['7576682', '7576692', '7576674', '7576667'] 2024-03-01T03:53:19.511 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-01T03:53:19.512 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}', 'tasks/{0-from/reef/{v18.2.1}'] 2024-03-01T03:53:19.512 INFO:scrape: 2024-03-01T03:53:19.512 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-03-01T03:53:19.512 INFO:scrape:1 jobs: ['7576711'] 2024-03-01T03:53:19.512 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-03-01T03:53:19.512 INFO:scrape: 2024-03-01T03:53:19.512 INFO:scrape:Dead: None 2024-03-01T03:53:19.512 INFO:scrape:1 jobs: ['7576695'] 2024-03-01T03:53:19.512 INFO:scrape: 2024-03-01T03:53:19.512 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-01T03:53:19.512 INFO:scrape:1 jobs: ['7576665'] 2024-03-01T03:53:19.512 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-03-01T03:53:19.513 INFO:scrape: 2024-03-01T03:53:19.515 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/ failed: 30 dead: 1 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [7576661] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/cap-flush} ----------------------------------------------------------------- time: 00:11:59 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576661/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576661/ SELinux denials found on ubuntu@smithi005.front.sepia.ceph.com: ['type=AVC msg=audit(1709220552.067:210): avc: denied { checkpoint_restore } for pid=1214 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576665] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:02:50 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576665/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576665/ error during scrub thrashing: rank damage found: {'backtrace'} [7576667] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:22 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576667/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576667/ reached maximum tries (51) after waiting for 300 seconds [7576671] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:41:30 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576671/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576671/ SELinux denials found on ubuntu@smithi123.front.sepia.ceph.com: ['type=AVC msg=audit(1709221073.830:199): avc: denied { checkpoint_restore } for pid=1200 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576672] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:15:34 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576672/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576672/ SELinux denials found on ubuntu@smithi129.front.sepia.ceph.com: ['type=AVC msg=audit(1709221057.248:200): avc: denied { checkpoint_restore } for pid=1206 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576673] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:38:09 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576673/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576673/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd57b0e95134cd89980416730e47fa4 Command failed (workunit test suites/fsx.sh) on smithi038 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7576674] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:09 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576674/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576674/ reached maximum tries (51) after waiting for 300 seconds [7576675] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/journal-repair} ----------------------------------------------------------------- time: 00:22:54 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576675/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576675/ SELinux denials found on ubuntu@smithi107.front.sepia.ceph.com: ['type=AVC msg=audit(1709221200.844:211): avc: denied { checkpoint_restore } for pid=1230 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576678] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:24:17 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576678/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576678/ SELinux denials found on ubuntu@smithi001.front.sepia.ceph.com: ['type=AVC msg=audit(1709221250.184:208): avc: denied { checkpoint_restore } for pid=1206 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576679] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:47 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576679/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576679/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d41a24bac1f4465b357553a42464847 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7576680] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:29:01 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576680/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576680/ SELinux denials found on ubuntu@smithi008.front.sepia.ceph.com: ['type=AVC msg=audit(1709221315.016:199): avc: denied { checkpoint_restore } for pid=1210 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576681] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:23:47 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576681/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576681/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7576682] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:47:01 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576682/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576682/ reached maximum tries (51) after waiting for 300 seconds [7576685] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:39:29 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576685/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576685/ SELinux denials found on ubuntu@smithi188.front.sepia.ceph.com: ['type=AVC msg=audit(1709221835.937:208): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576687] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:51:01 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576687/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576687/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ea080946cd44c8983f0a4a89b649456 Command failed (workunit test suites/dbench.sh) on smithi064 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7576690] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} ----------------------------------------------------------------- time: 00:40:00 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576690/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576690/ SELinux denials found on ubuntu@smithi082.front.sepia.ceph.com: ['type=AVC msg=audit(1709222246.943:198): avc: denied { checkpoint_restore } for pid=1154 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576692] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:38 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576692/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576692/ reached maximum tries (51) after waiting for 300 seconds [7576694] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:17:20 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576694/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576694/ SELinux denials found on ubuntu@smithi005.front.sepia.ceph.com: ['type=AVC msg=audit(1709222411.166:212): avc: denied { checkpoint_restore } for pid=1381 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576697] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:48 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576697/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576697/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25cd5714231c4608bfd50e886fd0fff3 Command failed (workunit test suites/fsx.sh) on smithi114 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7576700] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:41:08 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576700/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576700/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f67c124611b7434c818d791507d31dba Command failed (workunit test suites/fsstress.sh) on smithi079 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee923917d2c5fa780715183f495e7aa4da771e6e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7576704] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 11:43:11 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576704/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b926023ad9984c33b742cdbdff1fc19a Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7576705] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:19:48 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576705/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abfca2e5bee94c5781f848c56395e04c Command failed on smithi057 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1aac9824-d71e-11ee-95c3-87774f69a715 -e sha1=6085b91fefeb74bf4310344a87d81d3217ef7893 -- bash -c 'ceph orch ps'" [7576706] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 03:08:10 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576706/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576706/ valgrind error: Leak_StillReachable malloc malloc strdup [7576708] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:11:03 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576708/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576708/ SELinux denials found on ubuntu@smithi184.front.sepia.ceph.com: ['type=AVC msg=audit(1709223094.001:210): avc: denied { checkpoint_restore } for pid=1229 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576709] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:51:03 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576709/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576709/ SELinux denials found on ubuntu@smithi177.front.sepia.ceph.com: ['type=AVC msg=audit(1709223194.679:210): avc: denied { checkpoint_restore } for pid=1210 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1', 'type=AVC msg=audit(1709223194.679:209): avc: denied { checkpoint_restore } for pid=1206 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576710] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:19:21 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576710/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576710/ SELinux denials found on ubuntu@smithi012.front.sepia.ceph.com: ['type=AVC msg=audit(1709223004.978:212): avc: denied { checkpoint_restore } for pid=1341 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576711] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:46:11 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576711/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576711/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6cb9d43952b4853a6fec682de6ff10e Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7576712] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:26:00 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576712/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576712/ valgrind error: Leak_StillReachable malloc malloc strdup [7576716] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:48:59 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576716/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576716/ SELinux denials found on ubuntu@smithi002.front.sepia.ceph.com: ['type=AVC msg=audit(1709223333.849:199): avc: denied { checkpoint_restore } for pid=1229 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576718] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:26:41 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576718/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576718/ sentry: https://sentry.ceph.com/organizations/ceph/?query=45b759643e3e43e4aec06b348bb75fb7 Command failed on smithi116 with status 110: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 04eef8aa-d71f-11ee-95c3-87774f69a715 -e sha1=6085b91fefeb74bf4310344a87d81d3217ef7893 -- bash -c 'ceph orch ps'" Dead ================================================================= [7576695] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snapshots} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576695/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576695/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7576659] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:12:25 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576659/ [7576660] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:38 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576660/ [7576662] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:22:50 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576662/ [7576663] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:40:23 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576663/ [7576664] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:19 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576664/ [7576666] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-recovery} time: 00:59:33 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576666/ [7576668] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:27:06 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576668/ [7576669] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:29:54 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576669/ [7576670] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 01:06:09 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576670/ [7576676] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} time: 01:21:08 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576676/ [7576677] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:19 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576677/ [7576683] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 03:27:31 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576683/ [7576684] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 01:03:32 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576684/ [7576686] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:59:27 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576686/ [7576688] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:23:42 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576688/ [7576689] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 01:24:03 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576689/ [7576691] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:37:51 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576691/ [7576693] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:46 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576693/ [7576696] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:14:31 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576696/ [7576698] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:17:05 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576698/ [7576699] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:44:43 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576699/ [7576701] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:05:43 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576701/ [7576702] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/test_journal_migration} time: 00:15:21 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576702/ [7576703] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} time: 00:15:21 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576703/ [7576707] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} time: 00:16:06 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576707/ [7576713] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:20:10 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576713/ [7576714] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:01 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576714/ [7576715] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 02:29:50 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576715/ [7576717] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:43:41 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576717/ [7576719] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 00:57:42 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:53:23-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/7576719/