2023-08-31T18:25:32.499 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-08-31T18:25:32.501 INFO:teuthology.results:subset: '131/512' 2023-08-31T18:25:32.501 INFO:teuthology.results:seed: '6543' 2023-08-31T19:33:23.504 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-08-31T19:33:23.526 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-01T07:10:12.162 INFO:teuthology.results:Tests finished! gathering results... 2023-09-01T07:10:26.733 INFO:scrape:Found 101 jobs 2023-09-01T07:10:45.371 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.372 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.372 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.373 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.473 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.474 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.475 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.475 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.575 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.576 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.576 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.577 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.677 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.678 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.678 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.679 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.783 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.784 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.784 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.785 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.884 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.885 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.885 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.886 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.984 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.985 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.985 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:45.985 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.084 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.085 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.085 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.086 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.186 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.187 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.187 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.188 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.289 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.290 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.290 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.291 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.388 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.389 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.389 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.389 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.487 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.488 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.488 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.489 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.590 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.590 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.591 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.591 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.693 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.714 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.715 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.715 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.812 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.813 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.814 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.814 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.915 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.915 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.916 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:46.916 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.020 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.021 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.022 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.022 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.121 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.121 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.122 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.122 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.221 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.222 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.222 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.223 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.321 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.322 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.322 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.323 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.427 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.428 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.428 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.429 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.529 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.529 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.530 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.530 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.629 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.630 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.630 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.631 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.727 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.728 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.728 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:47.728 WARNING:scrape:Saw end of BT but not start 2023-09-01T07:10:53.462 WARNING:scrape:Misunderstood line: 2023-09-01T07:10:57.508 WARNING:scrape:Misunderstood line: 2023-09-01T07:11:01.867 INFO:scrape:Found 34 distinct failure reasons 2023-09-01T07:11:01.868 INFO:scrape:Failure: Command failed on smithi040 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" 2023-09-01T07:11:01.868 INFO:scrape:15 jobs: ['7385783', '7385777', '7385791', '7385768', '7385798', '7385812', '7385822', '7385817', '7385765', '7385825', '7385840', '7385849', '7385851', '7385860', '7385831'] 2023-09-01T07:11:01.869 INFO:scrape:suites intersection: ['centos_8', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-09-01T07:11:01.869 INFO:scrape:suites union: ['1-client', '1-mount/mount/fuse', '1-tests/dbench}}}', '1-tests/fsstress}}}', '1-tests/iozone}}}', '2-upgrade', '2-workload/stress_tests/{0-client-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_8', 'clusters/1-mds-2-client-micro', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus', 'tasks/{0-octopus'] 2023-09-01T07:11:01.869 INFO:scrape: 2023-09-01T07:11:01.870 INFO:scrape:Failure: 2023-09-01T07:11:01.870 INFO:scrape:30 jobs 2023-09-01T07:11:01.871 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-09-01T07:11:01.871 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-common}}', '1-tests/fscrypt-iozone}}', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/pjd}}', 'bluestore-bitmap', 'centos_latest', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/alternate-pool}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/data-scan}', 'tasks/failover}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/scrub}', 'tasks/trim-i24137}', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/snaps}', 'tasks/{0-client', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'traceless/50pc}'] 2023-09-01T07:11:01.872 INFO:scrape: 2023-09-01T07:11:01.872 INFO:scrape:Failure: Host key for server 'smithi098.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAcNfltreHCFkT3rx4tfIVepEtck0V35rZ9/vfsVOZQU', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5mhDdXQvkY2aZ3RQbRWhp7uU0gSVRersMVQh9Ht0+kfvOB4J6d9d3FwCSs2uqpflMjNZOMh0i13y2gT0vHxV9i8KZNga0BqYDqXDBY8CXp6C7yZ58UjA+fiR2gAxuD1Sv4UbLHrRhxrnhs/Y3NLm+ZiXGHCkenr3cRFnk+27CZic2BPu84eGeFYRQBTZSwD43LUVln4M1YUFckPQLzCK9anguoq1Mm2kLYdVizNAJdEPtLLXVtos24LGN+q4xCtJTyKn/12xZ6Nnsyg8Z40e+39cQcTFgFtaW0kFhsYnXUFEcG/FO8GEZ5bqpXNJk5TDO8hscrre6jVkXCbfwWEVjF05zgnt783+bSTuCABATnJbGzsycJeu1H9fFco6TBM9vpIGSsmLIwBKKRVHr0RbBQp3Ng76TfNIbk4MVfB++e825qMqUtRXn6uuodG4fKONTHjASGlQg2NnK3jtvwyipwvwtOGi7ahr8MDl2kZ7uyrDiNcxD3BEeuccaUzKNKk=' 2023-09-01T07:11:01.873 INFO:scrape:1 jobs: ['7385796'] 2023-09-01T07:11:01.873 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/admin}'] 2023-09-01T07:11:01.873 INFO:scrape: 2023-09-01T07:11:01.874 INFO:scrape:Failure: Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHCAubYhee5fRENuM1NddpCLTwWJMYW98idIokDIeE2l', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCk/Vs+scxYc9dKvsc9FiDi8PExUbbqiczaEvJ1zjcfjPZPQg2QkaLQRJlRpojwXB9U350+U+Z6h4aRat/gPUP/6srydLk5nm9tqC6+Lalq0ZCQk11OkvkfrBgFafgzYWSlJ7OZEp8ciDiwsd2noRKAvEbEPyjq+bhxTKrjYEjWQzXQmRKyQZY6tUMqw5SrtAdLCrskz6SE2mYxYhQlwflKsBgX+UHF2IiyEo7DxSgGKU6jnGZ3gxOVK5yAFypCbH9lZLxDvky7ZKptj0fegZEf8eUq0IA1VjMDpILr7kzwhipBRAPzZd0196eolwsy6RsEZwVlm3+HgxtXbKUAHU0PtQAUFJhOLbq5IEumfwQbv+CDV80IzgGmf2k7BVwLRqcYh0YAkQPit+bIHvSDyGSsfI9tXoJImhep9p9LgmNV1JHCo0t3Hi+TfkY6YdWcFAXXt161Dn5wlaK4bkMeFngAsiJCSYXSllPauIJx3OnD0A1qv6RpVhFHSbQeCwqMOk0=' 2023-09-01T07:11:01.874 INFO:scrape:1 jobs: ['7385786'] 2023-09-01T07:11:01.875 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-01T07:11:01.875 INFO:scrape: 2023-09-01T07:11:01.875 INFO:scrape:Failure: "2023-08-31T18:47:30.474023+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-09-01T07:11:01.876 INFO:scrape:5 jobs: ['7385762', '7385839', '7385811', '7385810', '7385769'] 2023-09-01T07:11:01.876 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-01T07:11:01.877 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/fstop}', 'tasks/metrics}', 'tasks/mirror}}', 'tasks/pool-perm}'] 2023-09-01T07:11:01.877 INFO:scrape: 2023-09-01T07:11:01.877 INFO:scrape:Failure: Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIO3S93IzQbwSip/ubhFIOZRH9a2C5nkTm5wa8uL72bMd', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3H2sgU5DmZvXJ7qyVSvCoARHSQHg2xk0tnbrHqYjs8GCuC4GPFr0sRUbLaugOTPNtIpLHD6bStWTNtnHrdbpL0PS6+OSpQIBEo2NIzhBux++DWGtYdU+BYT1SetsnVnDjN1SQyt+Z1IQEa7KjK6vF04CJy79rQeosQRXoBO89ykEAqspr3KtnvfCvdjJlv2//bOU52h3cURYK9xK3NYG+yHyjGKdPOkn5fJRTgXvA2P30qdQOzoICY11H5RipFcMsZIcVwTQe1QtxeK3TIi1hIDD2OAWBdLKTEnXFIcwFLSm9oA9JeLxyI+0nlVcLn9rFLIBWTuGiOT6qTlThYrDbMwpfkH3szSEQiUDyX9ug7xh9nTRsldvYOSlXfjGWbqNXGO0tFQIw4jw0gh+DmQbnE+DeEJJxrld1wBNTFgjAYXKcPwbZPbvDN/DaAlBB+h1Pne9m94We1lgcF6R5DxyJ75aHUqhfUir7iimHiS8bspxOiTzv3GdngWZNyzxoUik=' 2023-09-01T07:11:01.878 INFO:scrape:1 jobs: ['7385806'] 2023-09-01T07:11:01.878 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-01T07:11:01.879 INFO:scrape: 2023-09-01T07:11:01.879 INFO:scrape:Failure: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKP+YUeq+szP5RhxQUo+wFX+38voGCwwhPMYU1VgjHQO', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRlbWQIXAJ+6BdnF8RzS6zN5dJTJUdd9TCVjPiaP2P6JvN1X2+zZ7gtVeDh7uzkzV27qLSQ+q/CUSKt9wcgIIJCA8Fq14tp6uZl+nJlv9WpZk69xF4sCFa+hJx2TVTLiy8x1w6I2D4TDCTbaTLQpl6Grd/NI0O+IQ9KBV4LJJt15q15EbCtcH7qciS6bjFRjwyq6wxdbBt6ZgXaL9IGCgqVDbpwBJTkxrL67qNfrp4BX/pWCPDyII56RVlM8c5+Sh0wnsvg+R2jDHd+4xGtGMIEmpufhaxIwON8LUQIukqlrSvrgiQKJDPOssKOfmhbXuzOTsDq2dUt8zECB9VFSg/fl3YMd0lTxTJyuAs9L+QJoE7KrYRvxlUM874qYUpVT4ADw4u+2ftXrGld7VdhFZG4ssp2aJr9DW89Z6ir/nZi/6tzEocEFqISY7ncv8hTQfOoZgXlaxkFwxzAfDtHjh8wSviRYKdGtV/2b3/nMls6/q2qq1p5yGyaBe4dG5F5gE=' 2023-09-01T07:11:01.879 INFO:scrape:1 jobs: ['7385784'] 2023-09-01T07:11:01.880 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_fsstress', 'traceless/50pc}'] 2023-09-01T07:11:01.880 INFO:scrape: 2023-09-01T07:11:01.880 INFO:scrape:Dead: b'2023-09-01T06:59:31.921 DEBUG:teuthology.exit:Finished running handlers' 2023-09-01T07:11:01.881 INFO:scrape:3 jobs: ['7385790', '7385775', '7385789'] 2023-09-01T07:11:01.881 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-01T07:11:01.882 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/multifs-auth}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-09-01T07:11:01.882 INFO:scrape: 2023-09-01T07:11:01.883 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-09-01T07:11:01.883 INFO:scrape:2 jobs: ['7385764', '7385848'] 2023-09-01T07:11:01.883 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-09-01T07:11:01.884 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-09-01T07:11:01.884 INFO:scrape: 2023-09-01T07:11:01.885 INFO:scrape:Failure: Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIInSh7EpaRKGIrgUF4W9b/3cfi6VRsd7y80D+EfJFSaD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDEmz7TnZnnnyaax2fptOqvotowYD0f1z+9c27TkjMBZYdgs/yuX85JP+Y1k7nuHVBWRZ1xqqfeX+mC5qUPoGkWwrFm1kOwn6T8YhIXm/nPy6nqqC34Hqy6v0nOmDeDQiPCbjIAuqqxU7MZ0euQV1xqjp6n3EtpwrUnpVZgmEQy/nSSxk0WZZdR0lclQ1w/A40izkLYAaQ14QdUqzbxwmwa//egDnh8qUzpfEnGJ5OWAOAxdSxiBaiCVjN/FDd6r6K3LacrDF1YWEXqFg+3MmMnuLCJJjcHgp6lYVyNCq0rJyqdjzVASIXu1xMCOmDfD7qYMpIskxvWz/mjW4Phmb/7biilaAlA7AO5Iw6ewG2a4iK7NWWbe4KdztAQt5rOPfUed2e9knv07VR1O13yMfO/7VfofvjPNLNOec/VjcwXa1cD6WPMP13v9PiAYMdq2WyJ2AJLgPPqefHF1GhNZmOS2NeA0GSP1j5TT20SVT4q5yyEyV0slgWfXDw6oOZO+80=' 2023-09-01T07:11:01.885 INFO:scrape:1 jobs: ['7385788'] 2023-09-01T07:11:01.885 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-09-01T07:11:01.886 INFO:scrape: 2023-09-01T07:11:01.886 INFO:scrape:Failure: Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILVUmDOxSDXpmUjPmXbdGPqfTZDS2NCe/XzdvJWcGXJ+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRvIrRQYC4QhHeTo9oPoaSk/G5/IF/ntU2TCjQet4aABbjWAPrLmtuz8WMv6JAQBIPNJzp5tX9GMwGhE/dnvNaHdC03ZkbIIpARTvoR0D9QWhM022uRWMfeWLS9Umiz5zxrqY+wnmbd68DRY2eW2y9qho5tME6w7qWfBd0Z11VvURhYSQOOSqW9CPgb/HOcEMPXfgW3PErCa1l22Tiq5q4ur/dxmumrqe1mYLs8hzbDo8Wtc2KQobhDtKFkxDA0wolW7G7XGt5mqBaLrLqYjQ3MPzSIpi7XMbvUGLV7eQoXhMY/2b0f1QB80re41LpmBkDcwU2syGcHwoj5+kyjly8+A1FLkQ7DkmQuryI8ck20DdqJxxhtDUcvSACMKLOOG6EXnNB42VqRKrYpOr9weqZOs2C1pGZ3YISpz2f+aqiO7mVCiVaV1SFRhBu9tlf7Z4dZ9upybH0DwLhsz411gQNFchUpWB0T+ofSR5xV5fj4QgiCqlk7OVJ+H2aTtm13pk=' 2023-09-01T07:11:01.887 INFO:scrape:1 jobs: ['7385771'] 2023-09-01T07:11:01.887 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-01T07:11:01.887 INFO:scrape: 2023-09-01T07:11:01.888 INFO:scrape:Failure: Host key for server 'smithi053.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK234+DKAmG6EU4bwcJOPzPZtfFnTFxKfhZ/87klx9Fm', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDE3/etabK62/0YsVnvhgsjs/r9EW9yJeSFetiQQqtA2c25SPrmuNm7WEqdBaQxn7zSy12dPxDcMl+o3cYbVNV9ZVoiWQ8F5S6/G5l4dMBMxzpeb1L7YGm3uHauSk4Dttsjb0veoMOBhKPzTSdVgdB1uDLmEXvnQ5Va7oRJ/z0xWnr0aEL3A/JLFZXcCqIgsFV2q8ZDShPCouhXLhHKqziOkTQ3lQAAhUdg8ix3piJwHuSeP6zK9NSwkHKPZ0nR0SCAdsFs936CzlvVzjjEm01nRPJ5GUWyDSPUBfobDBFMHzzrw9Y5nSHJjPf2cncIlIjiKIRfVhV72j7VdZPdsGm5av+jwJYVyNo8xwPhvRIKI1UtWJgl8E1qDWnU5+5OzahHDeXZ8rLyykb8DUs8b5Bc3lERUKCXiXTQne3yiYf9QIDvFOv+5JsVu4OJ5POIYMp0bmbSdrZVxMe0Iwtf5BpEJMpjkgBCP3S1pihKCPeCQiJRPZzcKh22/X/xhsvIcjM=' 2023-09-01T07:11:01.888 INFO:scrape:1 jobs: ['7385774'] 2023-09-01T07:11:01.889 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-01T07:11:01.889 INFO:scrape: 2023-09-01T07:11:01.889 INFO:scrape:Failure: Host key for server 'smithi114.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEz5Xqq4aQNhjfUOw2pUCXWQUdCNqS5P5de/2TF/jrNe', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCt26F9VrhdZZif6fAuLU9AM3UUju4Hh9J1pQzvTxSEdikYS1W/tiMixZoFNlFZC2NrMRToHwIZ0ODiOaDZdAayibgEkOHV8hpOFohzoWfiLCvOVywUA2Df5nfTd79IIFV1Om365RszeTEkIuKNpAcXDrL6z9OjH7SZjJNiD8tjlhmjZBGrEikInBsVRDf+vTJWH1Dan25QrYxGBY5aip8nnqxRNnIONZYdMLX8tglkMlOpel8knWmkdC+ZaJZ4lFDPKn4NniwdHisNa392CRxQ2JoUz8Yfm2pRyoaZ3Yb3buCq9R27ZZoV6YY4AOJVOS9ZfmSSvzR9ki14+Y0A3AbzSTrrOPgo0BPVVnd1Co/ZHk2Ea3k1k1rxviA+bUe1AVPxioIRE3GcISWqr5NQ2BjQRvTGpwW+3GmUWP9X+eqCO6/cIOtcUNIR37WThfj8gdMkzulrfKpPjaO/XVjSX2yqPDq6TEOMNhSIidjxaut+loNaDZMH7onHYSxqoJh1tbk=' 2023-09-01T07:11:01.890 INFO:scrape:1 jobs: ['7385805'] 2023-09-01T07:11:01.890 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}'] 2023-09-01T07:11:01.891 INFO:scrape: 2023-09-01T07:11:01.891 INFO:scrape:Failure: Host key for server 'smithi194.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKO7LOOAiVzwo50p7dqfOINcxrheQoDMiSvOQlyom0uD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDg3T7S2XzDKWywt2nCebKeUVip1hiXN0AHVgonBSW/BG372QR5GAzvDa8PA2GVRp1TXTihaeuQscoYJUiAxoG3wXoaBlBFnVtvaA73R6glvRVGmC/3CreI2JCwitG8UkndurqOAd2owk8ex4ro3vH37Q0xhg8eYuvQmTgfhsGW0pjc7MOaYqbo/XgWBikEaQ0xSJ7365TqcIDihSz9b8/w2lWgLNHDOZVUISsYfNA2J65kYKNZEiNVLRUK8sPBWXM/cpXmfCXZhK3jRo9tHk2g8tguA9J5alvOfvJqXixxToGDu3UZOBF7C/mRmoY1AMM4Tl8h/VQiCabaNjgzeqbif0GCyTC3XvJjpG/qCtFMp4HF/5JZ56C+6+VcFYO0aMnmyXyC2ekD1J1iNrf+IVQQHL6Owe1xqPm3JXNTezn+Z4Ok3JLKjtAL19qN3i01rAE33CkMEm41Uj9Z9jwXS2eKg7aiwxMifieEgt2OuQH+5cnmYbyY2bV2Ez5IKQlCNOc=' 2023-09-01T07:11:01.891 INFO:scrape:1 jobs: ['7385852'] 2023-09-01T07:11:01.892 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-09-01T07:11:01.892 INFO:scrape: 2023-09-01T07:11:01.892 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-09-01T07:11:01.893 INFO:scrape:1 jobs: ['7385781'] 2023-09-01T07:11:01.893 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-09-01T07:11:01.893 INFO:scrape: 2023-09-01T07:11:01.894 INFO:scrape:Failure: Host key for server 'smithi117.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIulQwZrBxlnlBQVcFuZLDV6bHdXaSx5Hptl+5wWVe4q', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDIxt5aAXYEPe0M+BayWVuVusnQAmevxw8p1bSsjnmkprRi1w+Q0n/6wn2vvOr6ndirvHB0jQlFcEwitsDp5TKxBGplWUPOcuQFJK5ZfehoJIby+IYgufX4n70Wd9pTP7JOT9eWYXyGLtQfk0+cgMXpEVvWKih3/e4k0TfsqDcVPb0vaa+W1Y8C5CpRfl1flqqt/gohUtY3l01KesKPlbbeLtBBFN5H6lvQFKncwE9CQ2qRO9PZBySD6nnjAlUjNsfj9uJH9MZ5OZThM9dA4E97fo46bWWZntQ6NJ7fpHdQa9Q9FjS+oQtwQ5SMUlXrxOuMI5OGJic1CrhY4157qGFeO5al1WiXE9LutjbV2dShQKsMc9mBtQ1wW40wkvfdQN2j+Sc5OqstbsIRC7lpmAWwUEbvoJ7JL8vhBqmlcoRlKbVhQMmhcxZu6PerLiNuoBPkF9IkQPBxGaHflXg33TrLu23Iaym+E5e2d0jQYv5sD+J1sFQ/YaZ17lOLCDH6BLM=' 2023-09-01T07:11:01.894 INFO:scrape:1 jobs: ['7385856'] 2023-09-01T07:11:01.895 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-01T07:11:01.895 INFO:scrape: 2023-09-01T07:11:01.895 INFO:scrape:Failure: Host key for server 'smithi148.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIC7s4UxZ/jRydnBvHYCNTscWaVg9ZgZXNn0xX7DmjrmS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCWOm8GU0XieDcsLViEDe029DtlDts7ORo31WmZwcictF8wM+mCPqI2RoEjglzTEvvwij57xu/lyF0O+WF1MJIilvFBN1trBmsFfdZkSHJMVenYuYWyr2D3mpuGIQKGiXZ3B9iMeV2arPkiyEchqDSCeiXb3EoMf4kB7b3vFx+2V4AfJ+PydK3kRLIFlDTMXp4IYVHp1w8YpsxoSJBeay8fPKvDCPEoSbkcQ9TWO5XfZPJGXS4qx3E266Kby5lfYgl6phLzorF6YMws7EwyXmKDimKDJaQl3sx1wabt8ljmb+8LcuOjOf8tBugHctMrfr8Bk6HBn3LoMx7fSIBhuYTaEUJfUTq+DHQkca08gBqQBxKM//IDL3E8YQJlycmLYLseX8vKibt16vgGX8b6H9ZwXoO0h39HOYEFTQlJQKF8Ryj7HkFr6753qhnKCWuU/dfDp2mDxY5G4P5Q0FWDhSMJ1iXgze0BO2W9iBL9EEmSKxbbjmD3KZDHj1e7tTF1FlU=' 2023-09-01T07:11:01.896 INFO:scrape:1 jobs: ['7385809'] 2023-09-01T07:11:01.896 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-09-01T07:11:01.896 INFO:scrape: 2023-09-01T07:11:01.897 INFO:scrape:Failure: Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILkAfjP3osPysiTPOFivMHFW/pdr5hVioGAFm8skYUi+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC+cmVs+nH/xEr1czf38ZwS6SbnKZunSfx0TBF2rH+h5xNxpjhgHbPkx7lAD5E2VNErSUPlYs8tHXxTBd/YbX6jAwrXFGdlp/cMojzAiO3PQYDijFOTf3FCLRHqe+R/GNPgufuwOcZFln1NbQMM/nLvdXc+7b/1GrblkLxD9drG7Dfue9U/nBe/7g79eY8i1QI7HXZqTTQhDNzmm4ZjPinWDB2DQOL1t0FfrDkIVgzM7wYN2lNyKowbeMl1EEBFmbPKt6HxpB0Ig0fEh/2NtqFfrB7bGc2vHoE0F/84Hj62VScTTBoBN2XHF6FE8gnSMoP8Rd3ywLJZNLY+mtY1bY+R5WbLGWP2J3/l0XX5A4F4N3GbJiphgasQ78dP4strWRVnFMUnVgGR7kWT+SX8g99iTFr9Bp5N93HRUbas8tMCCM89L1sNDIO7G7rv4+TZn5JV/5Z6YtJAqS35vJYGmHQ3VFY0NnzVgclXabeMiUB+ML24jtbjAhcWYitZO/MNy6U=' 2023-09-01T07:11:01.897 INFO:scrape:1 jobs: ['7385833'] 2023-09-01T07:11:01.898 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-09-01T07:11:01.898 INFO:scrape: 2023-09-01T07:11:01.898 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f277b5568dd] 2023-09-01T07:11:01.899 INFO:scrape:ceph version 18.0.0-5814-gcc14fc8a (cc14fc8aa878d09ac353379f17aec3b92248b995) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f277ae7c520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f277b5568dd] 6: /usr/lib/ceph/libceph-common.so.2(+0x163a37) [0x7f277b556a37] 7: ceph-mds(+0x2c1e4b) [0x55e5dd087e4b] 8: (MDCache::split_dir(CDir*, int)+0x421) [0x55e5dd090831] 9: ceph-mds(+0x121dbd) [0x55e5dcee7dbd] 10: (CommonSafeTimer::timer_thread()+0x146) [0x7f277b63a3f6] 11: /usr/lib/ceph/libceph-common.so.2(+0x247e71) [0x7f277b63ae71] 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f277aeceb43] 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f277af60a00] 2023-09-01T07:11:01.899 INFO:scrape:1 jobs: ['7385826'] 2023-09-01T07:11:01.899 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-09-01T07:11:01.900 INFO:scrape: 2023-09-01T07:11:01.900 INFO:scrape:Failure: Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) 2023-09-01T07:11:01.900 INFO:scrape:1 jobs: ['7385823'] 2023-09-01T07:11:01.901 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] 2023-09-01T07:11:01.901 INFO:scrape: 2023-09-01T07:11:01.902 INFO:scrape:Failure: Host key for server 'smithi129.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAlcPvZVC9BTx38UBguSjbQvZRmQw3oA6UDG7JNK8+BQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCkaJSQR9w9I08MpmGHzY4R2qyskhB9DDKHasc0xKP15DZGHES94wwMehgeD/VtZu2i9CXyi934zk39q3q1+YKGzhUKC7EXB0XDB+5xpY5DUjviaT60sn0TMF9BpmJZZl2dILfr3KDd6mHY1BQ0NrIzn9OZeVuTr+I3E0uGy/G085YT+kRGhzmEjEnn183Sth7kQ7QCVmdqRJMFqxWHzDjk4PvAo5+qdmLOYXthLZ71JWtjN1UqJjsJp8hk29hLENfuwQTMh0hNMTu9/y/iMmfViFWb4TWYpEDRj/4yMvTTX2ZoulObTuFf4PaISL7iIb3btlYOlOqA45bDjxtxWfPqxwSjTRAVQOpD5JcqYQJuQACUPB2LmdFPf5KD0cLayfFC/92a6VqXUMA4zSWu02seuOLxpqMtoZHc0SAiQYWWs3dNzTrjHdZrGSyf2/tDVBT1qya5Fnci2RWmGadVP//yIaQXZ2rp7xCbFxsK9tUcRDvFD94TCnfjI7gso1V/qJs=' 2023-09-01T07:11:01.902 INFO:scrape:1 jobs: ['7385847'] 2023-09-01T07:11:01.902 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-01T07:11:01.903 INFO:scrape: 2023-09-01T07:11:01.903 INFO:scrape:Failure: Host key for server 'smithi040.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEilx+FPja+Zv7c7MDm8VeqUSgQ2KvYLlYI7U95ozla4', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRdkyFa7Qw5ReQ+NZ/gigXfKrHP6vRTL1pKXdZyQDToWNDgsJZd1aDu57Hn8IhPAsojDReK5y4Wej5/rzEMaYvlqi3O6eERvujm1tVdlyJlsOzFrG5G7oLZajcnHoaucS+C374Jjoq2ObbTRFWriVVlG31ZqOuPr1mNoqMNEaiLSCn4wfuo4EM+8t2C8/VqFpfqnd42VKcSg9nAU/X9W9aPi/DgBsFxNeT+YkZ3zQ7HvBh1fORx3UEL37vlCYBMI2CUYcLjoqzPbfJzApIDHVjptpAYQ4bkdbCXY3Y6Fg74xpysEXLnSwqTETcfRvG8g7jx3bqvOGyJAglPPuwYTPzyR04UskQBkAM6+oDqXpxRIbfptRSdbsKU88IoI3DgIXWzH4AFpQKQSwHZRZ9l+ujoobIeJpfrjUMAC7rZdzRIJLGSQpkRLuedKDpAP/BvleQMoAYewOGuquhBB78ICls6f1uEFbOIRbCcChRN8cqSYQBOolaUPHDjv3PMO7m+zc=' 2023-09-01T07:11:01.904 INFO:scrape:1 jobs: ['7385767'] 2023-09-01T07:11:01.904 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_ffsb', 'traceless/50pc}'] 2023-09-01T07:11:01.904 INFO:scrape: 2023-09-01T07:11:01.905 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-09-01T07:11:01.905 INFO:scrape:1 jobs: ['7385838'] 2023-09-01T07:11:01.905 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-09-01T07:11:01.906 INFO:scrape: 2023-09-01T07:11:01.906 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-09-01T07:11:01.907 INFO:scrape:1 jobs: ['7385850'] 2023-09-01T07:11:01.907 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-01T07:11:01.907 INFO:scrape: 2023-09-01T07:11:01.908 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-01T07:11:01.908 INFO:scrape:2 jobs: ['7385782', '7385819'] 2023-09-01T07:11:01.908 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-01T07:11:01.909 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-01T07:11:01.909 INFO:scrape: 2023-09-01T07:11:01.910 INFO:scrape:Failure: Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIMSR7pht4nCxqzO1T3+ecxhkWphfi6mHKmJIYOZqPz1b', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC5br7Zd4l6Y9BKt6q68NkvwSQcnvFHNFSBpRs3gvpSm0xvYHfRgLxYZpT0oYJmD1cepeywsCRv8YCAIQHAo0WBFWkXJdeKdL0LEdkNlCdnKza8io3OI8IrvNLDsOnFb7WsH+DXq0RQOjdiCFfw4LGKJhwQq06b6GVWGiGUVNMk2nflKrM+rAv0kSvs58s37TrpU7BQOih4iJMYrbD72BUVM/nURQa9Q+Ugzzp7dvgYP5Meq1UjFhvCAai0+tZlZCb8mvug1CM64GDMl4uL7quO+NRwEfdo/88++Z2AqEOsDwGgKB/hDBCPAQUJ4b/kMUzBYYcLccjDMBm4Bh7kZyFprX1Upq2DJ36DZ/d+AHwNrcxEQnOn0OCIQsEBiZFlFLGDYN1kl1+k2U8DSLhmtoTtjigiEF+gEvZTqfSZMcWeJ6jlNo4Ll3/gkT8dvgQLv9YJTDUuv1VEl8u7Nh9me28Z4iqjGQVG8XldpYAMQQ2e8E2tJhZf/ofIkJfEl3D/Nrk=' 2023-09-01T07:11:01.910 INFO:scrape:1 jobs: ['7385855'] 2023-09-01T07:11:01.910 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{test}}'] 2023-09-01T07:11:01.911 INFO:scrape: 2023-09-01T07:11:01.911 INFO:scrape:Failure: Host key for server 'smithi055.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPPxG4D4Td7ufK5hc1yERQ4pLHi5nns67Lr9jb1cJ+SS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCwabvKPS7tAGrd5ZkD33G59UHJ+1Is67W6Lm8hv98d0W553BEnICD+wnyalvU51Kzr19pmRFuRPmOpocK3vTOpI81P92tOIKRRlfkHJ29d1uNTAmItS2X1ke1P9SkJ6/wPh/JSX1lCwZt7dWIWABzfloctqq23OVs7BBEQWza1sPmDSItbu2CsrZXyxFJBTL0apNwu8dEj6yPscMTKwWbUbJvE6PZyeUpW9YDu9jZunAGr30SU+h5+2znx2VmzgRCYH+K7b/PiqgH2qlEulj2wADWMt5RSol7+Lhtzh8c6vQn8ROE72MEW8e93Mhfcg22t98/37RGwTicMa9b0YOR85sUs9pRAZAeLGqpmuAUkXsnDgTH6XkiMAXWfTonfUmLI0R/W33hG6pKSGNcrOxLaE4aSVzPdz3r0QvFbpYF/lQgIwZjjYF/G09htPu7rvu2XdLJ4m62fnWwQVBbtiAzcyoBRRhmifOKTibVE4DbmvP8KRrq1ykt1zIa/pFfoOk8=' 2023-09-01T07:11:01.911 INFO:scrape:1 jobs: ['7385832'] 2023-09-01T07:11:01.912 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-01T07:11:01.912 INFO:scrape: 2023-09-01T07:11:01.913 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-09-01T07:11:01.913 INFO:scrape:1 jobs: ['7385780'] 2023-09-01T07:11:01.913 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-09-01T07:11:01.914 INFO:scrape: 2023-09-01T07:11:01.914 INFO:scrape:Failure: Host key for server 'smithi006.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIP5u3vyR0pPa+2+b76h1+f606vWFg392AeD4RMOhAZ6W', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDSvWYbqj+2rJRSiXk9+ASdhBdjS5QQMm6pu3HWomDUvPwdfxonVZUkJFAn87XVyINeGn41gtMzmND6U8OrguFV3fAlev42w+2L1UqrteVfVFmfdet7Rmsc05sHCwU4gQClkbDwUjy6Hctaj/wRMW7Nkfvp3HkPpx99TTZ02ltwQrTVB+2hAaa/FH3ytj4MbZT4EIyndI/cdVhyVKQmP5EdkyJTHv5h18M4HmCKu5gb1w8iMxkHadIffTf8ZmdY1K26jXckgtv4MP3aw7YsVM8V4raxs26FJ81ftr/qkBg5IDkFFiGyUQGrh4o5VKKhkZFi9ztZysfASdnv19T0cZPN0wwgYlMyo5DiZY7TFQWybQtQtDwtMgM6WPJQRj+gQjGK7jpqMqr2s1DNi5b+fHNu4PnljZWUSIfQIyqfXE9fG/CX9/jeGM2Cz8uxjVqFSp3zncaTwOH2Wm2X0bhRonNlhehu56MkgwJt/OmiH/hKkP4Lu0fo+E8d0YcdCFsHlt0=' 2023-09-01T07:11:01.914 INFO:scrape:1 jobs: ['7385772'] 2023-09-01T07:11:01.915 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2023-09-01T07:11:01.915 INFO:scrape: 2023-09-01T07:11:01.915 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh' 2023-09-01T07:11:01.916 INFO:scrape:1 jobs: ['7385794'] 2023-09-01T07:11:01.916 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-09-01T07:11:01.917 INFO:scrape: 2023-09-01T07:11:01.917 INFO:scrape:Failure: Host key for server 'smithi094.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIItTZaeKCT147PXQsv0OtDBS+QrdaADSBftGNjK/D4Cz', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCTlsUhPnyqH3OaeM8NNllq+isIifotBi0dk8TkoSxoerOhhfIYbLbrNc5fvcRGhPjRplSGow++pe/EiWOBmG8ovYKtYQQWCf6CZdCLH0q00aPxi/T2zdtnzeD6HrVmT9MI1cXJ3SwKvnZ9IgSH1meJRqJsBMybcjHeAoXMKQtLfT7yAc//VOTtXPgQM/ecaPxBr9whfwZxmgta6aiRN4W57XMRJv/+tT/8MGz2SnZxo2nRax6kx8bJYeVajjnlJ7xVrmKYhfJFbz0qfs95SCuXeyWPyhVrtjRxd4pYbOY6Mrz4q1Knb8ZKSkysWYAQxTqJEJZAlMTxr/VehUr0J5JJgzQwWJyWX6G+211xHTkglbc5ZVjAXK1g+n81hSn85VSuHbHiTy7cEInkeR/hXWVjWbe8Vow2S+6HIu+JHxxzdGKV6bgQsm2nr3wj7SAzN0HOYivZo/kPTQuiVVIJEawnhy/8xEzz4BjzbC+gsjKEJiHyprmzs48uf56ryg8JO1E=' 2023-09-01T07:11:01.917 INFO:scrape:1 jobs: ['7385820'] 2023-09-01T07:11:01.918 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-01T07:11:01.918 INFO:scrape: 2023-09-01T07:11:01.918 INFO:scrape:Failure: Host key for server 'smithi007.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICWWRtGBG7Nlk1RUzeDaF1O/gZmxBQQG5EtmgFn2lak3', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCvOmhgI1Yrw62u7NN15KtYynlvNUr0jebqCTwM6cOLMwufMYEeVxIKzhwjA4t6j/p4qwXtz5B9Ytq7nxNhdmD1xIgUvyPmC8u2AlqQdgb1yex/zaZmMUAYy6hjetRvdfvVUtwxWRCJfjAhvueXdWY3L4NsFMTcG3eRhmxi1xPQsjx27tMX/u8xeCTH9Vd9g5F+8Ysjf+bVKNS2NYa+ZP+pg1VCmvYmmvobrFtahW0xAE0d4pO1sBAHR8t5a957iPRSvghbjNV3gr+s+C5k/ky/sQMfuVmaJah1cAninOFTz+Y0NfQg9eBCYzNcuFliwOPJaC+S+jSen1VFT1paI9bHTWWomFBcOm9j4vcRfYmBqMeNw23c2cVIvKUX7V9D4TUhwLERjIZhnO201lJLcSaxrkYiqlDO6ZDAnJzk49CRfcFWnUu3UuARk/wcsRr6HWS9ItcJ6qxqN9lMwbZhBpdH+BZ/aTjERFbsji6lBxemt3heVT74GLSfbOLWU34LcBU=' 2023-09-01T07:11:01.919 INFO:scrape:1 jobs: ['7385836'] 2023-09-01T07:11:01.919 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-01T07:11:01.919 INFO:scrape: 2023-09-01T07:11:01.920 INFO:scrape:Failure: Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINdtGK+uES/7R8RbYRPnhDofx0ekRlZltxC0aQP3JwPh', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDZeUsf6mEAG2dxGx2PIVTJdhnDQ/sbqeiUvSA+dCdBZcqJEyNrwV3UnZYGeYsNbjgIKCfHy7AVe5k0QE1D1LPwoAlLFSAJkXqSHTPZfLkhVaHAPxoXJ54HA/qw/i2CIko9I9Ygb+pYOOMvn0qt5NIuF78agWsKJgGi6ZNRozagL8eS5VjdWuUJb/rHF4dh7+PRIH+xNnV1dA4zcVHyNQGZrZHszeOW4mU3EDbIhxoR3jBhPsfCkmfKnaowGrNL/oT9UJBMtNoI2f0IBHsL1N7aK3hS9IXt1SMBbS5aJNxiEpXlQhDIfJ63JslY/aYiwBwfI7mHqx7jh+tRVC4DhVmmqyyNdHUjomn9SAdfCvXd478B5NRBtH+yo+CGzkOMY15fDMDHhzDFGtYfKaE47UCacdDFElzTeeNMZCHsDDyAactOsIr28WaOv5xv+bV9ZbOZO97lxhYJPCCyFowFrG4Zcr6W+DGEoaNASEsjNAdUC3AfgRuuz6NVeOXFArg5lNs=' 2023-09-01T07:11:01.920 INFO:scrape:1 jobs: ['7385800'] 2023-09-01T07:11:01.920 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-09-01T07:11:01.921 INFO:scrape: 2023-09-01T07:11:01.921 INFO:scrape:Failure: Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILYVxOzxcSmp6jEtj9JZlPUJfCPOhw273OR0kIqiWUPt', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCzziTVYzyd3/2BlCMhFA1faC8VSweAsRyBInCOhQyxZOdDusj3+yB/awCBvZA0+dgqyrJ3CPdc0CSmpwcIyDIt0Xl3Sb7aOsKrePSsPwt7ZDrqAa7doIGKa8WNoyEFF0BMhutSFBV/i0pVBnHUj3U/1XG9R2ilJaGypq0j3IKR7p2ShLKCm90e7pE2iJLc3uw+y64okr3ZOc3XCu1jPx5Z7ypn5QNmo5MVk3q4aK8kLGsR5nJ1ZKFP6kUoM/CCJ6qqGWz5DGB2wVuVuNEzDFQKzDQRrLcQknUcQEeS4um/Oy432nXn2Ng1rbAo2z/eZT/56TPtxTib+dPoHwJoRSActdOwOUOF3Qw6PkDVYvJ50YEXGyFWi3dLAEsmxRrEn8XF2kaOCG9rwIwqBN0IJ/wl5ZadBoAr8QHSMpeCxQD00j3IjB8Js7FyYMxC8Adkl/JsUw0i2m9OM+goV5Py0FuVTTgVS8tYgZFcwwE6Hw5a00mibsqD4vp/q+nEicEWZ7M=' 2023-09-01T07:11:01.922 INFO:scrape:1 jobs: ['7385804'] 2023-09-01T07:11:01.922 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-09-01T07:11:01.922 INFO:scrape: 2023-09-01T07:11:01.929 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/ failed: 82 dead: 3 running: 0 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [7385760] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385760/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=818ba7f3474948b79ca79955304a849c [7385762] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:15:22 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385762/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385762/ "2023-08-31T18:47:30.474023+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385764] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:15:06 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385764/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3197afcc44364968869767818564ba84 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7385765] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} ----------------------------------------------------------------- time: 00:22:16 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385765/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385765/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e86040b67aec407d8d5e751ea412d43a Command failed on smithi094 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385766] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385766/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6b5b75df1484790a4556d96f66b6332 [7385767] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385767/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de1ff3e2ee554c05a2886bb8a1723c51 Host key for server 'smithi040.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEilx+FPja+Zv7c7MDm8VeqUSgQ2KvYLlYI7U95ozla4', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRdkyFa7Qw5ReQ+NZ/gigXfKrHP6vRTL1pK XdZyQDToWNDgsJZd1aDu57Hn8IhPAsojDReK5y4Wej5/rzEMaYvlqi3O6eERvujm1tVdlyJlsOz FrG5G7oLZajcnHoaucS+C374Jjoq2ObbTRFWriVVlG31ZqOuPr1mNoqMNEaiLSCn4wfuo4EM+8t 2C8/VqFpfqnd42VKcSg9nAU/X9W9aPi/DgBsFxNeT+YkZ3zQ7HvBh1fORx3UEL37vlCYBMI2CUY cLjoqzPbfJzApIDHVjptpAYQ4bkdbCXY3Y6Fg74xpysEXLnSwqTETcfRvG8g7jx3bqvOGyJAglP PuwYTPzyR04UskQBkAM6+oDqXpxRIbfptRSdbsKU88IoI3DgIXWzH4AFpQKQSwHZRZ9l+ujoobI eJpfrjUMAC7rZdzRIJLGSQpkRLuedKDpAP/BvleQMoAYewOGuquhBB78ICls6f1uEFbOIRbCcCh RN8cqSYQBOolaUPHDjv3PMO7m+zc=' [7385768] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:26:09 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385768/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e13a30a624be4451a384654f35764119 Command failed on smithi087 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385769] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:16:30 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385769/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385769/ "2023-08-31T18:51:10.084918+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385771] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385771/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385771/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d61a35f7a12c40399f393e99c8ccc4e8 Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILVUmDOxSDXpmUjPmXbdGPqfTZDS2NCe/XzdvJWcGXJ+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRvIrRQYC4QhHeTo9oPoaSk/G5/IF/ntU2T CjQet4aABbjWAPrLmtuz8WMv6JAQBIPNJzp5tX9GMwGhE/dnvNaHdC03ZkbIIpARTvoR0D9QWhM 022uRWMfeWLS9Umiz5zxrqY+wnmbd68DRY2eW2y9qho5tME6w7qWfBd0Z11VvURhYSQOOSqW9CP gb/HOcEMPXfgW3PErCa1l22Tiq5q4ur/dxmumrqe1mYLs8hzbDo8Wtc2KQobhDtKFkxDA0wolW7 G7XGt5mqBaLrLqYjQ3MPzSIpi7XMbvUGLV7eQoXhMY/2b0f1QB80re41LpmBkDcwU2syGcHwoj5 +kyjly8+A1FLkQ7DkmQuryI8ck20DdqJxxhtDUcvSACMKLOOG6EXnNB42VqRKrYpOr9weqZOs2C 1pGZ3YISpz2f+aqiO7mVCiVaV1SFRhBu9tlf7Z4dZ9upybH0DwLhsz411gQNFchUpWB0T+ofSR5 xV5fj4QgiCqlk7OVJ+H2aTtm13pk=' [7385772] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385772/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385772/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3a64978b8794cfebd488586fc4071c6 Host key for server 'smithi006.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIP5u3vyR0pPa+2+b76h1+f606vWFg392AeD4RMOhAZ6W', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDSvWYbqj+2rJRSiXk9+ASdhBdjS5QQMm6pu 3HWomDUvPwdfxonVZUkJFAn87XVyINeGn41gtMzmND6U8OrguFV3fAlev42w+2L1UqrteVfVFmf det7Rmsc05sHCwU4gQClkbDwUjy6Hctaj/wRMW7Nkfvp3HkPpx99TTZ02ltwQrTVB+2hAaa/FH3 ytj4MbZT4EIyndI/cdVhyVKQmP5EdkyJTHv5h18M4HmCKu5gb1w8iMxkHadIffTf8ZmdY1K26jX ckgtv4MP3aw7YsVM8V4raxs26FJ81ftr/qkBg5IDkFFiGyUQGrh4o5VKKhkZFi9ztZysfASdnv1 9T0cZPN0wwgYlMyo5DiZY7TFQWybQtQtDwtMgM6WPJQRj+gQjGK7jpqMqr2s1DNi5b+fHNu4Pnl jZWUSIfQIyqfXE9fG/CX9/jeGM2Cz8uxjVqFSp3zncaTwOH2Wm2X0bhRonNlhehu56MkgwJt/Om iH/hKkP4Lu0fo+E8d0YcdCFsHlt0=' [7385773] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385773/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c77a621e9f447e0af9a09a25c7aa89d [7385774] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385774/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfc798af6ec44be187d8ee2feecd869f Host key for server 'smithi053.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK234+DKAmG6EU4bwcJOPzPZtfFnTFxKfhZ/87klx9Fm', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDE3/etabK62/0YsVnvhgsjs/r9EW9yJeSFe tiQQqtA2c25SPrmuNm7WEqdBaQxn7zSy12dPxDcMl+o3cYbVNV9ZVoiWQ8F5S6/G5l4dMBMxzpe b1L7YGm3uHauSk4Dttsjb0veoMOBhKPzTSdVgdB1uDLmEXvnQ5Va7oRJ/z0xWnr0aEL3A/JLFZX cCqIgsFV2q8ZDShPCouhXLhHKqziOkTQ3lQAAhUdg8ix3piJwHuSeP6zK9NSwkHKPZ0nR0SCAds Fs936CzlvVzjjEm01nRPJ5GUWyDSPUBfobDBFMHzzrw9Y5nSHJjPf2cncIlIjiKIRfVhV72j7Vd ZPdsGm5av+jwJYVyNo8xwPhvRIKI1UtWJgl8E1qDWnU5+5OzahHDeXZ8rLyykb8DUs8b5Bc3lER UKCXiXTQne3yiYf9QIDvFOv+5JsVu4OJ5POIYMp0bmbSdrZVxMe0Iwtf5BpEJMpjkgBCP3S1pih KCPeCQiJRPZzcKh22/X/xhsvIcjM=' [7385776] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385776/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c1f4e24c1c4441b8c6b21ac525b1142 [7385777] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:26:32 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385777/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cfc7af330db2420c85d4974a3b31c2e7 Command failed on smithi012 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385779] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385779/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b39e1f4d2c174ab6b2424438dc07557d [7385780] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:12:24 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385780/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2ea431856914973821fbbe45213e6fd Command failed (workunit test fs/test_python.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7385781] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:31:45 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385781/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59f7f844bb2f434f97191c94fa3c90df Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) [7385782] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:24:46 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385782/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a8d8eed931140559dfc27a738dd3f17 saw valgrind issues [7385783] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:26:50 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385783/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6928967272d4788b369c2b58f8fc401 Command failed on smithi040 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385784] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385784/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29cb1b64eb2f48e88aa3ce8f396a6b7c Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKP+YUeq+szP5RhxQUo+wFX+38voGCwwhPMYU1VgjHQO', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRlbWQIXAJ+6BdnF8RzS6zN5dJTJUdd9TCV jPiaP2P6JvN1X2+zZ7gtVeDh7uzkzV27qLSQ+q/CUSKt9wcgIIJCA8Fq14tp6uZl+nJlv9WpZk6 9xF4sCFa+hJx2TVTLiy8x1w6I2D4TDCTbaTLQpl6Grd/NI0O+IQ9KBV4LJJt15q15EbCtcH7qci S6bjFRjwyq6wxdbBt6ZgXaL9IGCgqVDbpwBJTkxrL67qNfrp4BX/pWCPDyII56RVlM8c5+Sh0wn svg+R2jDHd+4xGtGMIEmpufhaxIwON8LUQIukqlrSvrgiQKJDPOssKOfmhbXuzOTsDq2dUt8zEC B9VFSg/fl3YMd0lTxTJyuAs9L+QJoE7KrYRvxlUM874qYUpVT4ADw4u+2ftXrGld7VdhFZG4ssp 2aJr9DW89Z6ir/nZi/6tzEocEFqISY7ncv8hTQfOoZgXlaxkFwxzAfDtHjh8wSviRYKdGtV/2b3 /nMls6/q2qq1p5yGyaBe4dG5F5gE=' [7385785] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385785/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385785/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1c4b8b464a240dfaf1dd17c37bbd4fd [7385786] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385786/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385786/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64306e939960418aa24058cca9962209 Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHCAubYhee5fRENuM1NddpCLTwWJMYW98idIokDIeE2l', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCk/Vs+scxYc9dKvsc9FiDi8PExUbbqiczaE vJ1zjcfjPZPQg2QkaLQRJlRpojwXB9U350+U+Z6h4aRat/gPUP/6srydLk5nm9tqC6+Lalq0ZCQ k11OkvkfrBgFafgzYWSlJ7OZEp8ciDiwsd2noRKAvEbEPyjq+bhxTKrjYEjWQzXQmRKyQZY6tUM qw5SrtAdLCrskz6SE2mYxYhQlwflKsBgX+UHF2IiyEo7DxSgGKU6jnGZ3gxOVK5yAFypCbH9lZL xDvky7ZKptj0fegZEf8eUq0IA1VjMDpILr7kzwhipBRAPzZd0196eolwsy6RsEZwVlm3+HgxtXb KUAHU0PtQAUFJhOLbq5IEumfwQbv+CDV80IzgGmf2k7BVwLRqcYh0YAkQPit+bIHvSDyGSsfI9t XoJImhep9p9LgmNV1JHCo0t3Hi+TfkY6YdWcFAXXt161Dn5wlaK4bkMeFngAsiJCSYXSllPauIJ x3OnD0A1qv6RpVhFHSbQeCwqMOk0=' [7385787] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385787/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69849a9e73c24ef686c0284f4f219960 [7385788] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385788/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e79e0670e6241f68d5d938fb86a018d Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIInSh7EpaRKGIrgUF4W9b/3cfi6VRsd7y80D+EfJFSaD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDEmz7TnZnnnyaax2fptOqvotowYD0f1z+9c 27TkjMBZYdgs/yuX85JP+Y1k7nuHVBWRZ1xqqfeX+mC5qUPoGkWwrFm1kOwn6T8YhIXm/nPy6nq qC34Hqy6v0nOmDeDQiPCbjIAuqqxU7MZ0euQV1xqjp6n3EtpwrUnpVZgmEQy/nSSxk0WZZdR0lc lQ1w/A40izkLYAaQ14QdUqzbxwmwa//egDnh8qUzpfEnGJ5OWAOAxdSxiBaiCVjN/FDd6r6K3La crDF1YWEXqFg+3MmMnuLCJJjcHgp6lYVyNCq0rJyqdjzVASIXu1xMCOmDfD7qYMpIskxvWz/mjW 4Phmb/7biilaAlA7AO5Iw6ewG2a4iK7NWWbe4KdztAQt5rOPfUed2e9knv07VR1O13yMfO/7Vfo fvjPNLNOec/VjcwXa1cD6WPMP13v9PiAYMdq2WyJ2AJLgPPqefHF1GhNZmOS2NeA0GSP1j5TT20 SVT4q5yyEyV0slgWfXDw6oOZO+80=' [7385791] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:27:15 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385791/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcaf62bac4594ef6a25552b5eb78df22 Command failed on smithi053 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385792] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385792/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385792/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54bd2f9b15064b72b8c0e87d2122c2a5 [7385793] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385793/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09bd95091a0d498291d6e97a379d910d [7385794] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:15:39 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385794/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385794/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15dd8f95a4cf42978ef756e9c81dea53 Command failed (workunit test fs/quota/quota.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh' [7385796] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385796/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef200d1f6ae541f59b8c6f07983e7aa2 Host key for server 'smithi098.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAcNfltreHCFkT3rx4tfIVepEtck0V35rZ9/vfsVOZQU', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5mhDdXQvkY2aZ3RQbRWhp7uU0gSVRersM VQh9Ht0+kfvOB4J6d9d3FwCSs2uqpflMjNZOMh0i13y2gT0vHxV9i8KZNga0BqYDqXDBY8CXp6C 7yZ58UjA+fiR2gAxuD1Sv4UbLHrRhxrnhs/Y3NLm+ZiXGHCkenr3cRFnk+27CZic2BPu84eGeFY RQBTZSwD43LUVln4M1YUFckPQLzCK9anguoq1Mm2kLYdVizNAJdEPtLLXVtos24LGN+q4xCtJTy Kn/12xZ6Nnsyg8Z40e+39cQcTFgFtaW0kFhsYnXUFEcG/FO8GEZ5bqpXNJk5TDO8hscrre6jVkX CbfwWEVjF05zgnt783+bSTuCABATnJbGzsycJeu1H9fFco6TBM9vpIGSsmLIwBKKRVHr0RbBQp3 Ng76TfNIbk4MVfB++e825qMqUtRXn6uuodG4fKONTHjASGlQg2NnK3jtvwyipwvwtOGi7ahr8MD l2kZ7uyrDiNcxD3BEeuccaUzKNKk=' [7385797] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385797/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385797/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f538fbdcf384b0a9f5e6258ecfa0a02 [7385798] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:24:14 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385798/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3ca2e2df3d14bb994fe68ebaf8d4c8b Command failed on smithi079 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385800] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385800/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e81ee72403f44757ac2ca433d4207161 Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINdtGK+uES/7R8RbYRPnhDofx0ekRlZltxC0aQP3JwPh', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDZeUsf6mEAG2dxGx2PIVTJdhnDQ/sbqeiUv SA+dCdBZcqJEyNrwV3UnZYGeYsNbjgIKCfHy7AVe5k0QE1D1LPwoAlLFSAJkXqSHTPZfLkhVaHA PxoXJ54HA/qw/i2CIko9I9Ygb+pYOOMvn0qt5NIuF78agWsKJgGi6ZNRozagL8eS5VjdWuUJb/r HF4dh7+PRIH+xNnV1dA4zcVHyNQGZrZHszeOW4mU3EDbIhxoR3jBhPsfCkmfKnaowGrNL/oT9UJ BMtNoI2f0IBHsL1N7aK3hS9IXt1SMBbS5aJNxiEpXlQhDIfJ63JslY/aYiwBwfI7mHqx7jh+tRV C4DhVmmqyyNdHUjomn9SAdfCvXd478B5NRBtH+yo+CGzkOMY15fDMDHhzDFGtYfKaE47UCacdDF ElzTeeNMZCHsDDyAactOsIr28WaOv5xv+bV9ZbOZO97lxhYJPCCyFowFrG4Zcr6W+DGEoaNASEs jNAdUC3AfgRuuz6NVeOXFArg5lNs=' [7385801] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385801/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64afa4f504b14608b01a460a983b4a41 [7385803] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385803/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385803/ sentry: https://sentry.ceph.com/organizations/ceph/?query=602016fb3c6348ddbbbd37ed3c055132 [7385804] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385804/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfc1b4e79c084936824bc3d54410f5ff Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILYVxOzxcSmp6jEtj9JZlPUJfCPOhw273OR0kIqiWUPt', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCzziTVYzyd3/2BlCMhFA1faC8VSweAsRyBI nCOhQyxZOdDusj3+yB/awCBvZA0+dgqyrJ3CPdc0CSmpwcIyDIt0Xl3Sb7aOsKrePSsPwt7ZDrq Aa7doIGKa8WNoyEFF0BMhutSFBV/i0pVBnHUj3U/1XG9R2ilJaGypq0j3IKR7p2ShLKCm90e7pE 2iJLc3uw+y64okr3ZOc3XCu1jPx5Z7ypn5QNmo5MVk3q4aK8kLGsR5nJ1ZKFP6kUoM/CCJ6qqGW z5DGB2wVuVuNEzDFQKzDQRrLcQknUcQEeS4um/Oy432nXn2Ng1rbAo2z/eZT/56TPtxTib+dPoH wJoRSActdOwOUOF3Qw6PkDVYvJ50YEXGyFWi3dLAEsmxRrEn8XF2kaOCG9rwIwqBN0IJ/wl5Zad BoAr8QHSMpeCxQD00j3IjB8Js7FyYMxC8Adkl/JsUw0i2m9OM+goV5Py0FuVTTgVS8tYgZFcwwE 6Hw5a00mibsqD4vp/q+nEicEWZ7M=' [7385805] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385805/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385805/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98dddd764df548a0ab99b515fa0452fa Host key for server 'smithi114.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEz5Xqq4aQNhjfUOw2pUCXWQUdCNqS5P5de/2TF/jrNe', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCt26F9VrhdZZif6fAuLU9AM3UUju4Hh9J1p QzvTxSEdikYS1W/tiMixZoFNlFZC2NrMRToHwIZ0ODiOaDZdAayibgEkOHV8hpOFohzoWfiLCvO VywUA2Df5nfTd79IIFV1Om365RszeTEkIuKNpAcXDrL6z9OjH7SZjJNiD8tjlhmjZBGrEikInBs VRDf+vTJWH1Dan25QrYxGBY5aip8nnqxRNnIONZYdMLX8tglkMlOpel8knWmkdC+ZaJZ4lFDPKn 4NniwdHisNa392CRxQ2JoUz8Yfm2pRyoaZ3Yb3buCq9R27ZZoV6YY4AOJVOS9ZfmSSvzR9ki14+ Y0A3AbzSTrrOPgo0BPVVnd1Co/ZHk2Ea3k1k1rxviA+bUe1AVPxioIRE3GcISWqr5NQ2BjQRvTG pwW+3GmUWP9X+eqCO6/cIOtcUNIR37WThfj8gdMkzulrfKpPjaO/XVjSX2yqPDq6TEOMNhSIidj xaut+loNaDZMH7onHYSxqoJh1tbk=' [7385806] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385806/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53931e84b9ba469493a0949715fee508 Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIO3S93IzQbwSip/ubhFIOZRH9a2C5nkTm5wa8uL72bMd', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3H2sgU5DmZvXJ7qyVSvCoARHSQHg2xk0tn brHqYjs8GCuC4GPFr0sRUbLaugOTPNtIpLHD6bStWTNtnHrdbpL0PS6+OSpQIBEo2NIzhBux++D WGtYdU+BYT1SetsnVnDjN1SQyt+Z1IQEa7KjK6vF04CJy79rQeosQRXoBO89ykEAqspr3KtnvfC vdjJlv2//bOU52h3cURYK9xK3NYG+yHyjGKdPOkn5fJRTgXvA2P30qdQOzoICY11H5RipFcMsZI cVwTQe1QtxeK3TIi1hIDD2OAWBdLKTEnXFIcwFLSm9oA9JeLxyI+0nlVcLn9rFLIBWTuGiOT6qT lThYrDbMwpfkH3szSEQiUDyX9ug7xh9nTRsldvYOSlXfjGWbqNXGO0tFQIw4jw0gh+DmQbnE+De EJJxrld1wBNTFgjAYXKcPwbZPbvDN/DaAlBB+h1Pne9m94We1lgcF6R5DxyJ75aHUqhfUir7iim HiS8bspxOiTzv3GdngWZNyzxoUik=' [7385807] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385807/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385807/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9ac04909eeb40488415c2c131015961 [7385808] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385808/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8cefc951bac9413a89fc6ecab8fe82aa [7385809] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385809/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd92dca2e4e242fbb5ed8dcee85f8db5 Host key for server 'smithi148.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIC7s4UxZ/jRydnBvHYCNTscWaVg9ZgZXNn0xX7DmjrmS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCWOm8GU0XieDcsLViEDe029DtlDts7ORo31 WmZwcictF8wM+mCPqI2RoEjglzTEvvwij57xu/lyF0O+WF1MJIilvFBN1trBmsFfdZkSHJMVenY uYWyr2D3mpuGIQKGiXZ3B9iMeV2arPkiyEchqDSCeiXb3EoMf4kB7b3vFx+2V4AfJ+PydK3kRLI FlDTMXp4IYVHp1w8YpsxoSJBeay8fPKvDCPEoSbkcQ9TWO5XfZPJGXS4qx3E266Kby5lfYgl6ph LzorF6YMws7EwyXmKDimKDJaQl3sx1wabt8ljmb+8LcuOjOf8tBugHctMrfr8Bk6HBn3LoMx7fS IBhuYTaEUJfUTq+DHQkca08gBqQBxKM//IDL3E8YQJlycmLYLseX8vKibt16vgGX8b6H9ZwXoO0 h39HOYEFTQlJQKF8Ryj7HkFr6753qhnKCWuU/dfDp2mDxY5G4P5Q0FWDhSMJ1iXgze0BO2W9iBL 9EEmSKxbbjmD3KZDHj1e7tTF1FlU=' [7385810] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:15:20 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385810/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385810/ "2023-08-31T19:21:53.856416+0000 mon.a (mon.0) 108 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385811] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:03:56 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385811/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385811/ "2023-08-31T19:24:13.947263+0000 mon.a (mon.0) 87 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385812] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:34:36 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385812/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77a4efda5bca44d9ba385841e6bc66a3 Command failed on smithi027 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385814] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385814/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13dd5b8baca04b9caaf30a74c996285a [7385815] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385815/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9120c59d6f7b4147b59950b6fc2429af [7385816] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385816/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14fcd93152f4484b93c230eb0237e727 [7385817] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:27:41 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385817/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5221b153bf5f44a29f4817f615a5e104 Command failed on smithi002 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385818] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385818/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4a105e619304a1ba4a51ad5cdd9c98c [7385819] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:26:17 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385819/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea32c3edfa384df588c5365b2cd8e6c1 saw valgrind issues [7385820] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385820/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be42910fdd3d43f28b57aaf4a7391e10 Host key for server 'smithi094.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIItTZaeKCT147PXQsv0OtDBS+QrdaADSBftGNjK/D4Cz', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCTlsUhPnyqH3OaeM8NNllq+isIifotBi0dk 8TkoSxoerOhhfIYbLbrNc5fvcRGhPjRplSGow++pe/EiWOBmG8ovYKtYQQWCf6CZdCLH0q00aPx i/T2zdtnzeD6HrVmT9MI1cXJ3SwKvnZ9IgSH1meJRqJsBMybcjHeAoXMKQtLfT7yAc//VOTtXPg QM/ecaPxBr9whfwZxmgta6aiRN4W57XMRJv/+tT/8MGz2SnZxo2nRax6kx8bJYeVajjnlJ7xVrm KYhfJFbz0qfs95SCuXeyWPyhVrtjRxd4pYbOY6Mrz4q1Knb8ZKSkysWYAQxTqJEJZAlMTxr/Veh Ur0J5JJgzQwWJyWX6G+211xHTkglbc5ZVjAXK1g+n81hSn85VSuHbHiTy7cEInkeR/hXWVjWbe8 Vow2S+6HIu+JHxxzdGKV6bgQsm2nr3wj7SAzN0HOYivZo/kPTQuiVVIJEawnhy/8xEzz4BjzbC+ gsjKEJiHyprmzs48uf56ryg8JO1E=' [7385822] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} ----------------------------------------------------------------- time: 00:34:30 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385822/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4834b64ce8f145039a5687c7db2ff04f Command failed on smithi082 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385823] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:45:19 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385823/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f845aa301912456ea2b1a38b2a0d6c91 Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) [7385824] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385824/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0290c40ea28b4b799d50dd5b6833689e [7385825] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:35:47 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385825/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4f8626c74634dd09d8cda8473f76495 Command failed on smithi029 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385826] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:37:42 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385826/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385826/ "2023-08-31T19:30:21.561588+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385827] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385827/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc25386ffc7c4f7884ad8090ab301562 [7385828] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385828/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=601f08ac1aeb424da5669ec1944bbff4 [7385829] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385829/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1cd1fbfeb2f4016b49199d01cd14070 [7385831] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:24:59 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385831/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385831/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2003532da77445f58aaf918d12368f40 Command failed on smithi003 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385832] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385832/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=135d17d968344cd3836b3d5412964564 Host key for server 'smithi055.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPPxG4D4Td7ufK5hc1yERQ4pLHi5nns67Lr9jb1cJ+SS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCwabvKPS7tAGrd5ZkD33G59UHJ+1Is67W6L m8hv98d0W553BEnICD+wnyalvU51Kzr19pmRFuRPmOpocK3vTOpI81P92tOIKRRlfkHJ29d1uNT AmItS2X1ke1P9SkJ6/wPh/JSX1lCwZt7dWIWABzfloctqq23OVs7BBEQWza1sPmDSItbu2CsrZX yxFJBTL0apNwu8dEj6yPscMTKwWbUbJvE6PZyeUpW9YDu9jZunAGr30SU+h5+2znx2VmzgRCYH+ K7b/PiqgH2qlEulj2wADWMt5RSol7+Lhtzh8c6vQn8ROE72MEW8e93Mhfcg22t98/37RGwTicMa 9b0YOR85sUs9pRAZAeLGqpmuAUkXsnDgTH6XkiMAXWfTonfUmLI0R/W33hG6pKSGNcrOxLaE4aS VzPdz3r0QvFbpYF/lQgIwZjjYF/G09htPu7rvu2XdLJ4m62fnWwQVBbtiAzcyoBRRhmifOKTibV E4DbmvP8KRrq1ykt1zIa/pFfoOk8=' [7385833] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385833/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=911b416f18484ba1be8673830904d9bd Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILkAfjP3osPysiTPOFivMHFW/pdr5hVioGAFm8skYUi+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC+cmVs+nH/xEr1czf38ZwS6SbnKZunSfx0T BF2rH+h5xNxpjhgHbPkx7lAD5E2VNErSUPlYs8tHXxTBd/YbX6jAwrXFGdlp/cMojzAiO3PQYDi jFOTf3FCLRHqe+R/GNPgufuwOcZFln1NbQMM/nLvdXc+7b/1GrblkLxD9drG7Dfue9U/nBe/7g7 9eY8i1QI7HXZqTTQhDNzmm4ZjPinWDB2DQOL1t0FfrDkIVgzM7wYN2lNyKowbeMl1EEBFmbPKt6 HxpB0Ig0fEh/2NtqFfrB7bGc2vHoE0F/84Hj62VScTTBoBN2XHF6FE8gnSMoP8Rd3ywLJZNLY+m tY1bY+R5WbLGWP2J3/l0XX5A4F4N3GbJiphgasQ78dP4strWRVnFMUnVgGR7kWT+SX8g99iTFr9 Bp5N93HRUbas8tMCCM89L1sNDIO7G7rv4+TZn5JV/5Z6YtJAqS35vJYGmHQ3VFY0NnzVgclXabe MiUB+ML24jtbjAhcWYitZO/MNy6U=' [7385836] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385836/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f10575361fa48b8a6a087c087d51d1f Host key for server 'smithi007.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICWWRtGBG7Nlk1RUzeDaF1O/gZmxBQQG5EtmgFn2lak3', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCvOmhgI1Yrw62u7NN15KtYynlvNUr0jebqC TwM6cOLMwufMYEeVxIKzhwjA4t6j/p4qwXtz5B9Ytq7nxNhdmD1xIgUvyPmC8u2AlqQdgb1yex/ zaZmMUAYy6hjetRvdfvVUtwxWRCJfjAhvueXdWY3L4NsFMTcG3eRhmxi1xPQsjx27tMX/u8xeCT H9Vd9g5F+8Ysjf+bVKNS2NYa+ZP+pg1VCmvYmmvobrFtahW0xAE0d4pO1sBAHR8t5a957iPRSvg hbjNV3gr+s+C5k/ky/sQMfuVmaJah1cAninOFTz+Y0NfQg9eBCYzNcuFliwOPJaC+S+jSen1VFT 1paI9bHTWWomFBcOm9j4vcRfYmBqMeNw23c2cVIvKUX7V9D4TUhwLERjIZhnO201lJLcSaxrkYi qlDO6ZDAnJzk49CRfcFWnUu3UuARk/wcsRr6HWS9ItcJ6qxqN9lMwbZhBpdH+BZ/aTjERFbsji6 lBxemt3heVT74GLSfbOLWU34LcBU=' [7385837] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385837/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef72badf27844d9f8c2f50fd35bad7f9 [7385838] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:54:15 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385838/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc1d1bbc59274fe2baccf34f05c117ee Command failed (workunit test suites/dbench.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7385839] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:32:25 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385839/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385839/ "2023-08-31T19:39:10.483793+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7385840] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:29:37 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385840/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e013bfefb85489cb2d394adfb2e43eb Command failed on smithi019 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385841] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385841/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8525b273725b435aa4159201397a75c5 [7385843] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385843/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ddbb2bbfddeb43609f75f2a65f9fbe3e [7385846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385846/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8cc14e44c3d41949ff97c33088ad192 [7385847] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385847/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd6812e0b36e4ce59bae6ae6afa28adc Host key for server 'smithi129.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAlcPvZVC9BTx38UBguSjbQvZRmQw3oA6UDG7JNK8+BQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCkaJSQR9w9I08MpmGHzY4R2qyskhB9DDKHa sc0xKP15DZGHES94wwMehgeD/VtZu2i9CXyi934zk39q3q1+YKGzhUKC7EXB0XDB+5xpY5DUjvi aT60sn0TMF9BpmJZZl2dILfr3KDd6mHY1BQ0NrIzn9OZeVuTr+I3E0uGy/G085YT+kRGhzmEjEn n183Sth7kQ7QCVmdqRJMFqxWHzDjk4PvAo5+qdmLOYXthLZ71JWtjN1UqJjsJp8hk29hLENfuwQ TMh0hNMTu9/y/iMmfViFWb4TWYpEDRj/4yMvTTX2ZoulObTuFf4PaISL7iIb3btlYOlOqA45bDj xtxWfPqxwSjTRAVQOpD5JcqYQJuQACUPB2LmdFPf5KD0cLayfFC/92a6VqXUMA4zSWu02seuOLx pqMtoZHc0SAiQYWWs3dNzTrjHdZrGSyf2/tDVBT1qya5Fnci2RWmGadVP//yIaQXZ2rp7xCbFxs K9tUcRDvFD94TCnfjI7gso1V/qJs=' [7385848] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385848/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a28c016d5933409685a0e5d8adcb25e1 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7385849] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:28:22 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385849/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f3ce768419444458fd9595ef67e0402 Command failed on smithi057 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385850] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:41:28 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385850/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385850/ error during scrub thrashing: rank damage found: {'backtrace'} [7385851] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 00:28:59 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385851/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=90e11b2911424158b49f6659f2d7d7af Command failed on smithi094 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7385852] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385852/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385852/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3284b3ffa8e54c09b5d1cfd121e9318e Host key for server 'smithi194.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKO7LOOAiVzwo50p7dqfOINcxrheQoDMiSvOQlyom0uD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDg3T7S2XzDKWywt2nCebKeUVip1hiXN0AHV gonBSW/BG372QR5GAzvDa8PA2GVRp1TXTihaeuQscoYJUiAxoG3wXoaBlBFnVtvaA73R6glvRVG mC/3CreI2JCwitG8UkndurqOAd2owk8ex4ro3vH37Q0xhg8eYuvQmTgfhsGW0pjc7MOaYqbo/Xg WBikEaQ0xSJ7365TqcIDihSz9b8/w2lWgLNHDOZVUISsYfNA2J65kYKNZEiNVLRUK8sPBWXM/cp XmfCXZhK3jRo9tHk2g8tguA9J5alvOfvJqXixxToGDu3UZOBF7C/mRmoY1AMM4Tl8h/VQiCabaN jgzeqbif0GCyTC3XvJjpG/qCtFMp4HF/5JZ56C+6+VcFYO0aMnmyXyC2ekD1J1iNrf+IVQQHL6O we1xqPm3JXNTezn+Z4Ok3JLKjtAL19qN3i01rAE33CkMEm41Uj9Z9jwXS2eKg7aiwxMifieEgt2 OuQH+5cnmYbyY2bV2Ez5IKQlCNOc=' [7385853] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385853/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385853/ sentry: https://sentry.ceph.com/organizations/ceph/?query=418cea20ec92480082aa45f651a47d82 [7385855] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{test}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385855/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385855/ sentry: https://sentry.ceph.com/organizations/ceph/?query=86345afbf906467482c624543932827f Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIMSR7pht4nCxqzO1T3+ecxhkWphfi6mHKmJIYOZqPz1b', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC5br7Zd4l6Y9BKt6q68NkvwSQcnvFHNFSBp Rs3gvpSm0xvYHfRgLxYZpT0oYJmD1cepeywsCRv8YCAIQHAo0WBFWkXJdeKdL0LEdkNlCdnKza8 io3OI8IrvNLDsOnFb7WsH+DXq0RQOjdiCFfw4LGKJhwQq06b6GVWGiGUVNMk2nflKrM+rAv0kSv s58s37TrpU7BQOih4iJMYrbD72BUVM/nURQa9Q+Ugzzp7dvgYP5Meq1UjFhvCAai0+tZlZCb8mv ug1CM64GDMl4uL7quO+NRwEfdo/88++Z2AqEOsDwGgKB/hDBCPAQUJ4b/kMUzBYYcLccjDMBm4B h7kZyFprX1Upq2DJ36DZ/d+AHwNrcxEQnOn0OCIQsEBiZFlFLGDYN1kl1+k2U8DSLhmtoTtjigi EF+gEvZTqfSZMcWeJ6jlNo4Ll3/gkT8dvgQLv9YJTDUuv1VEl8u7Nh9me28Z4iqjGQVG8XldpYA MQQ2e8E2tJhZf/ofIkJfEl3D/Nrk=' [7385856] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385856/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fb2ae6dbaac4a25b20875f2b8b3459b Host key for server 'smithi117.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIulQwZrBxlnlBQVcFuZLDV6bHdXaSx5Hptl+5wWVe4q', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDIxt5aAXYEPe0M+BayWVuVusnQAmevxw8p1 bSsjnmkprRi1w+Q0n/6wn2vvOr6ndirvHB0jQlFcEwitsDp5TKxBGplWUPOcuQFJK5ZfehoJIby +IYgufX4n70Wd9pTP7JOT9eWYXyGLtQfk0+cgMXpEVvWKih3/e4k0TfsqDcVPb0vaa+W1Y8C5Cp Rfl1flqqt/gohUtY3l01KesKPlbbeLtBBFN5H6lvQFKncwE9CQ2qRO9PZBySD6nnjAlUjNsfj9u JH9MZ5OZThM9dA4E97fo46bWWZntQ6NJ7fpHdQa9Q9FjS+oQtwQ5SMUlXrxOuMI5OGJic1CrhY4 157qGFeO5al1WiXE9LutjbV2dShQKsMc9mBtQ1wW40wkvfdQN2j+Sc5OqstbsIRC7lpmAWwUEbv oJ7JL8vhBqmlcoRlKbVhQMmhcxZu6PerLiNuoBPkF9IkQPBxGaHflXg33TrLu23Iaym+E5e2d0j QYv5sD+J1sFQ/YaZ17lOLCDH6BLM=' [7385857] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385857/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec3f8cfcd0654891aea14f31451c056f [7385858] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385858/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=086f9f16a5904173b45c23723b530187 [7385859] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385859/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81dce6a9696d4b5b8f63daff046c374a [7385860] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:40:43 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385860/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31ad516a83be4fed87e61aa5148a529b Command failed on smithi162 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" Dead ================================================================= [7385775] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385775/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385775/ hit max job timeout [7385789] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385789/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385789/ hit max job timeout [7385790] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385790/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385790/ hit max job timeout Pass ================================================================= [7385761] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:39:27 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385761/ [7385763] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:32:55 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385763/ [7385770] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:39:00 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385770/ [7385778] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:27:12 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385778/ [7385795] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:47 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385795/ [7385799] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:48:14 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385799/ [7385802] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:31 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385802/ [7385813] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:36:36 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385813/ [7385821] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:15:40 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385821/ [7385830] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 01:39:02 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385830/ [7385834] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:36:15 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385834/ [7385835] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:17:14 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385835/ [7385842] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:47:11 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385842/ [7385844] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:29 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385844/ [7385845] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:41:05 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385845/ [7385854] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:14 info: http://pulpito.ceph.com/pdonnell-2023-08-31_16:24:38-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7385854/