2024-01-31T04:16:29.198 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-31T04:16:29.198 INFO:teuthology.results:subset: '293/1024' 2024-01-31T04:16:29.199 INFO:teuthology.results:seed: '9619' 2024-01-31T05:32:00.986 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-31T05:32:01.013 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-31T17:09:52.609 INFO:teuthology.results:Tests finished! gathering results... 2024-01-31T17:09:56.592 INFO:scrape:Found 49 jobs 2024-01-31T17:09:58.814 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539468/teuthology.log 2024-01-31T17:10:03.011 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.011 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.011 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.011 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.070 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.070 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.070 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.070 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.074 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.074 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.075 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.075 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.079 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.079 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.079 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.079 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.083 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.084 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.084 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.084 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.096 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.096 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.096 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.096 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.176 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.177 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.177 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.177 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.181 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.182 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.182 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.182 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.186 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.186 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.186 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.186 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.191 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.191 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.191 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.191 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.195 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.196 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.196 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.196 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.208 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.208 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.209 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.209 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.285 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.286 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.286 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.286 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.290 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.290 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.290 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.290 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.294 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.294 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.295 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.295 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.299 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.299 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.299 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.299 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.304 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.304 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.304 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.304 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.316 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.317 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.317 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.317 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.396 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.396 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.396 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.397 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.401 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.401 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.401 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.401 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.405 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.405 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.406 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.406 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.410 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.410 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.410 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.410 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.415 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.415 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.415 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.415 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.427 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.427 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.427 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.428 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.505 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.505 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.505 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.505 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.509 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.510 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.510 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.510 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.514 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.514 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.514 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.514 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.519 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.519 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.519 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.519 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.523 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.523 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.524 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.524 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.536 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.536 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.536 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.536 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.615 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.616 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.616 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.616 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.620 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.620 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.621 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.621 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.625 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.625 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.625 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.625 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.629 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.629 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.629 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.629 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.634 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.634 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.634 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.634 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.647 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.648 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.648 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.648 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.732 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.732 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.733 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.733 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.737 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.737 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.737 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.737 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.741 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.741 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.742 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.742 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.746 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.746 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.746 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.746 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.750 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.751 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.751 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.751 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.763 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.763 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.763 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.763 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.838 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.839 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.839 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.839 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.843 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.844 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.844 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.844 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.848 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.848 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.848 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.848 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.852 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.852 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.852 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.853 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.857 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.857 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.857 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.857 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.869 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.869 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.869 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:03.869 WARNING:scrape:Saw end of BT but not start 2024-01-31T17:10:17.361 INFO:scrape:Found 14 distinct failure reasons 2024-01-31T17:10:17.361 INFO:scrape:Failure: Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' 2024-01-31T17:10:17.362 INFO:scrape:28 jobs: ['7539466', '7539471', '7539491', '7539498', '7539460', '7539481', '7539493', '7539463', '7539465', '7539477', '7539502', '7539470', '7539485', '7539484', '7539488', '7539489', '7539490', '7539501', '7539487', '7539461', '7539469', '7539507', '7539505', '7539506', '7539478', '7539492', '7539476', '7539508'] 2024-01-31T17:10:17.362 INFO:scrape:suites intersection: [] 2024-01-31T17:10:17.362 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '2-workload/dbench', '3-inline/no', '3-inline/yes', '3-snaps/no', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-verify}', '5-upgrade-with-workload', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', '6-verify}}', 'begin/{0-install', 'centos_9.stream', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host}', 'export-check', 'fail_fs/no', 'fail_fs/yes', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/nfs/{cluster/{1-node}', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/ignorelist_health', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'roles', 'session_timeout}', 'standby-replay', 'supported-random-distros$/{ubuntu_latest}', 'syntax}', 'tasks/nfs}', 'tasks/{0-from/quincy', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-01-31T17:10:17.363 INFO:scrape: 2024-01-31T17:10:17.363 INFO:scrape:Failure: valgrind error: Leak_StillReachable malloc malloc strdup 2024-01-31T17:10:17.363 INFO:scrape:3 jobs: ['7539473', '7539480', '7539499'] 2024-01-31T17:10:17.363 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'mount/fuse'] 2024-01-31T17:10:17.363 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/ignorelist_health', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'tasks/mirror}}', 'ubuntu_latest}}', 'validater/valgrind}'] 2024-01-31T17:10:17.363 INFO:scrape: 2024-01-31T17:10:17.363 INFO:scrape:Dead: None 2024-01-31T17:10:17.363 INFO:scrape:3 jobs: ['7539468', '7539504', '7539496'] 2024-01-31T17:10:17.363 INFO:scrape: 2024-01-31T17:10:17.364 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2024-01-31T17:10:17.364 INFO:scrape:2 jobs: ['7539500', '7539475'] 2024-01-31T17:10:17.364 INFO:scrape:suites intersection: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-pacific'] 2024-01-31T17:10:17.364 INFO:scrape:suites union: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'fs/upgrade/nofs/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-pacific'] 2024-01-31T17:10:17.364 INFO:scrape: 2024-01-31T17:10:17.364 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2024-01-31T17:10:17.364 INFO:scrape:1 jobs: ['7539483'] 2024-01-31T17:10:17.364 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/xfstests-dev}'] 2024-01-31T17:10:17.364 INFO:scrape: 2024-01-31T17:10:17.364 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-01-31T17:10:17.365 INFO:scrape:1 jobs: ['7539464'] 2024-01-31T17:10:17.365 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/pool-perm}'] 2024-01-31T17:10:17.365 INFO:scrape: 2024-01-31T17:10:17.365 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2024-01-31T17:10:17.365 INFO:scrape:2 jobs: ['7539462', '7539482'] 2024-01-31T17:10:17.365 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-01-31T17:10:17.365 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{centos_9.stream', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-01-31T17:10:17.365 INFO:scrape: 2024-01-31T17:10:17.365 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=661946e7044723130c03627323f6a879a9848cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-01-31T17:10:17.365 INFO:scrape:1 jobs: ['7539495'] 2024-01-31T17:10:17.365 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-01-31T17:10:17.366 INFO:scrape: 2024-01-31T17:10:17.366 INFO:scrape:Dead: b'2024-01-31T17:02:18.161 DEBUG:teuthology.exit:Finished running handlers' 2024-01-31T17:10:17.366 INFO:scrape:1 jobs: ['7539494'] 2024-01-31T17:10:17.366 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2024-01-31T17:10:17.366 INFO:scrape: 2024-01-31T17:10:17.366 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-01-31T17:10:17.366 INFO:scrape:2 jobs: ['7539467', '7539479'] 2024-01-31T17:10:17.366 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay', 'wsync/no}'] 2024-01-31T17:10:17.366 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2024-01-31T17:10:17.366 INFO:scrape: 2024-01-31T17:10:17.367 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi115 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=661946e7044723130c03627323f6a879a9848cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-01-31T17:10:17.367 INFO:scrape:1 jobs: ['7539503'] 2024-01-31T17:10:17.367 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2024-01-31T17:10:17.367 INFO:scrape: 2024-01-31T17:10:17.367 INFO:scrape:Failure: Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) 2024-01-31T17:10:17.367 INFO:scrape:1 jobs: ['7539472'] 2024-01-31T17:10:17.367 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/centos_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-01-31T17:10:17.367 INFO:scrape: 2024-01-31T17:10:17.367 INFO:scrape:Failure: Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) 2024-01-31T17:10:17.367 INFO:scrape:1 jobs: ['7539486'] 2024-01-31T17:10:17.368 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}'] 2024-01-31T17:10:17.368 INFO:scrape: 2024-01-31T17:10:17.368 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-01-31T17:10:17.368 INFO:scrape:1 jobs: ['7539497'] 2024-01-31T17:10:17.368 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/ubuntu_latest', 'fs/shell/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}'] 2024-01-31T17:10:17.368 INFO:scrape: 2024-01-31T17:10:17.371 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/ failed: 44 dead: 4 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7539460] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:09:50 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539460/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31848a807db14d9398e6f0977f744643 Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539461] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:07 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539461/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b88fb2b7f8f4956a7a14bf137a2a97c Command failed on smithi006 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539462] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:16 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539462/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b430301d789a4479b141e2075750f01d Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7539463] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:28 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539463/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5f967994ea84ea29e8331411045989a Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539464] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/pool-perm} ----------------------------------------------------------------- time: 00:13:58 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539464/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9a11201f3204b21b798c4942adc682f Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7539465] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:06 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539465/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9f33e7f67a7749a2a07dbabe44dc1ef2 Command failed on smithi023 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539466] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:11:58 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539466/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=712e8ce495d649eab51971af72f3cfb9 Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539467] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:43:02 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539467/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539467/ error during scrub thrashing: rank damage found: {'backtrace'} [7539469] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:12:06 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539469/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=577c387e9b7243bf9fbb64b48ee7b65b Command failed on smithi083 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539470] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:12:02 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539470/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=233829a692334eb09d2e144f4b7fb9ce Command failed on smithi104 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539471] fs/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} ----------------------------------------------------------------- time: 00:10:25 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539471/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=505177676a3f4da1bac24097386f9e23 Command failed on smithi158 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539472] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:50:25 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539472/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539472/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b2851aad5d24940b4eea9a6cba8a889 Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts) [7539473] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:25:01 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539473/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539473/ valgrind error: Leak_StillReachable malloc malloc strdup [7539475] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:20:30 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539475/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da84dfa9c71740d1b97143c950e905b1 Command failed on smithi016 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7539476] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:11:56 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539476/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=415c9284065b4fd58456182675555acd Command failed on smithi060 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539477] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:09:16 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539477/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539477/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b712e663851a4b819ebd178398420fbb Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539478] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:11:57 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539478/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d52ed6e06c6847349d08c63f2583376e Command failed on smithi047 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539479] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:25:17 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539479/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539479/ error during scrub thrashing: rank damage found: {'backtrace'} [7539480] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:48:31 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539480/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539480/ valgrind error: Leak_StillReachable malloc malloc strdup [7539481] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:10:51 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539481/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fff4817e192c4176929187e807fe4c32 Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539482] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:09:49 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539482/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539482/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6329cb842d34a198029387bd0ec3159 Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7539483] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:14:31 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539483/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21bd76ff615d44acb861ec82052920ba Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7539484] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:11:59 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539484/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f732e7ca38f9416c85c3af5eb8d6e754 Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539485] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:11:36 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539485/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539485/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3083e480a5804b05a368acb3d0bfcf0b Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539486] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} ----------------------------------------------------------------- time: 00:39:57 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539486/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539486/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76e1cc5fe5d34a04a4ecc321ae38b37b Test failure: test_multiple_path_r (tasks.cephfs.test_admin.TestFsAuthorize) [7539487] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:14:13 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539487/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25088b3ea725481eb2ea76c0f0bd7611 Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539488] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:41 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539488/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539488/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41ada7cf1aae4ef8bbfa403c97a7ee61 Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539489] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:08:52 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539489/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8edc74ad0bed47f68b3047e252e3f452 Command failed on smithi081 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539490] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:06 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539490/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539490/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae65e6a8d5094777a065a350d40843c8 Command failed on smithi069 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539491] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:13:56 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539491/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff2f34b89f9b41fc8be5ed3b94da3606 Command failed on smithi031 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539492] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:09:43 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539492/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539492/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bb8ee5c47524e5087cdcea9b82b6667 Command failed on smithi167 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539493] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:09:32 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539493/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5b5b3831e4e4a5bb4e03c24ae19a48c Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539495] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:51:54 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539495/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d6a28534a334789bad205b1b525d50a Command failed (workunit test suites/dbench.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=661946e7044723130c03627323f6a879a9848cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7539497] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:13:11 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539497/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539497/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14d11fca24d24c6a8e9bf4a2ebe23fc9 Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) [7539498] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:09:16 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539498/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539498/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3bda5ef75ee47b08afef5b8560fb253 Command failed on smithi088 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539499] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:25:33 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539499/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539499/ valgrind error: Leak_StillReachable calloc calloc _dl_check_map_versions [7539500] fs/upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:19:54 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539500/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9d30b41f6fe4dd3ac4179bdd0191b12 Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' [7539501] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:47 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539501/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539501/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a00b09d4c8d42bc952efabc7b460dd7 Command failed on smithi046 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539502] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:20 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539502/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539502/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b648a44562f44202a02c2446792fad24 Command failed on smithi063 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539503] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:31:08 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539503/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02102fcf7dd14bd1a1d1e5746e262b08 Command failed (workunit test suites/fsx.sh) on smithi115 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=661946e7044723130c03627323f6a879a9848cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7539505] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:35 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539505/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539505/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32f2e05f9ade41489c43227482ccc300 Command failed on smithi001 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539506] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:16:02 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539506/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8812dc72e0864dffb4c4c0ee1071faae Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' [7539507] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:10:03 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539507/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8375169f571a48c6a9bc17ab74661941 Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull' [7539508] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:26 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539508/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f2232f32f13475bb7407ad17e46276d Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:661946e7044723130c03627323f6a879a9848cb5 pull' Dead ================================================================= [7539468] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539468/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539468/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7539494] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539494/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539494/ hit max job timeout [7539496] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539496/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539496/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds [7539504] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539504/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539504/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7539474] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:35:39 info: https://pulpito.ceph.com/vshankar-2024-01-31_04:15:12-fs-wip-vshankar-testing-20240125.044502-testing-default-smithi/7539474/