2023-09-02T01:04:24.615 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-02T01:04:24.616 INFO:teuthology.results:subset: '131/512' 2023-09-02T01:04:24.616 INFO:teuthology.results:seed: '6543' 2023-09-02T01:40:53.764 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-02T01:40:53.768 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-02T13:35:46.370 INFO:teuthology.results:Tests finished! gathering results... 2023-09-02T13:35:54.950 INFO:scrape:Found 101 jobs 2023-09-02T13:36:01.326 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386769/teuthology.log 2023-09-02T13:36:17.609 WARNING:scrape:Misunderstood line: 2023-09-02T13:36:24.224 WARNING:scrape:Misunderstood line: 2023-09-02T13:36:39.855 INFO:scrape:Found 36 distinct failure reasons 2023-09-02T13:36:39.857 INFO:scrape:Failure: Host key for server 'smithi144.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICscwU18doVcjsD8alpnsyedzuCDTvDGIcKH4YiZTYka', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC0QePstQJ5rzFc9uN+Z4u9iOaiTSiNh5579frJCtuik0UickLwGb7bl8JH+PpncZI7StT25IdfU0hSiAYssJaNq3ru5apyajvphEGEcqRIRZd3ApgN+OXPDjvTSThSSSMQjYi4zFop8hVE1ZIq4/A7Ea5sI3nW09Nu5HzpVeBA2wpyvBjvw6QxljHmxaB6kiDU0BLKb7lUHoFy0t7uvH+7/kg1efdTEnnX8rj8KTaCz2JXZfhPNnwIzbG9+DmZROkLQdTLMGZWvSkdBHb9w8eiAr0FHUN5WVcyk4F2QjaEHbJuVv8mb56qJqJ3Mh8TbFZa13lVl3heoJBKVp4g/UF41hpWmCACH62vZJUNMj7NZDvUUixIR3g4iovVFgMWC+O7odhxFw9rMpwLdPJZko6k9GplHh/v0YFHk+id57mMMIYgJdy0yvxm5g7otvGQfNuDxPmFLCocZCcXxZ22MEzFI2Jboihzp/zmpp2TKBCbLRRjaMfV3MkCHHNIUae7pmk=' 2023-09-02T13:36:39.857 INFO:scrape:1 jobs: ['7386727'] 2023-09-02T13:36:39.858 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/admin}'] 2023-09-02T13:36:39.858 INFO:scrape: 2023-09-02T13:36:39.859 INFO:scrape:Failure: Host key for server 'smithi097.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIImUrwqDLKVvSGsEWfds752ApEoo4UucmgKJ+jzAhKtX', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQD0bdzCebIPvnj2DJYPXZG7oeitjm3x1jJXR4IifxxY9TqvHX6bc+Ao8x2kd39cif9jOJFHxCe24qewXEELaqD29r2hKRNSASCZB77ne8lDsfNgMZ4oaKd9pQr3h5bNqTXp7Tm92cilhFWfkavEZVGLAW0BsdjqQJlj5/fQ0jaIrJYZCpqdqQI2WcGFI2NeHSbmSK39pR5y7nxTMiOZmywAvFXMDEf8Ywuh/jjvtl0l//8QK8tXWpaITtvseLBLaDkvw5VICF1C+5x2wkJmxFEAXAm3xyz3vVGXu4BtHh8EeMF3p+2b0IzTNuGlhX6jKSEjLSvenqDkHzKsazwX3vcl4pAucmoztk30Fibx+rT5yjX/hcljnf9y4kQxYrurh7QO/mwH9yuG62gsCwPL8uF4CQJlk4HcwYuGpchwQfAIZZz0jsACF9R2Nd0s995u8d/wlCAqFPCp4pCH99kURMmXX/nwJWHgGqTnh3WGA9MTe3mouGdHs9Dvw+Ju6K5F2SE=' 2023-09-02T13:36:39.859 INFO:scrape:1 jobs: ['7386703'] 2023-09-02T13:36:39.859 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2023-09-02T13:36:39.860 INFO:scrape: 2023-09-02T13:36:39.860 INFO:scrape:Failure: Host key for server 'smithi088.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIN1TsfUbicrUG9DyfeQ88+tz3SBWlSCS5XggOx3Qkn1f', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDEARstn4P5i4DzlVnbubfEVC06kksZLPIvYa4b02Orm6MQWuMaqtoKOk/hbIDmWyM6xeRtNb0gdi64qcnmpwV4Wj1Zw+h+AQoP7oI0jaHIg8L8i0rpBojfkOblK52QV1umih/15rFzc/+xFn7JnxbyUiuZn2G5n4VMgpMEsmJV5K/XONYnt8nCnenSGXntdX5xfgJrlhIa+sD61MmXrm4ePteHy9MsKw9geuid+tpnJ10eI7jo6gS/wsfoWM9JTzjAxK1h7zBF3qFDutnnznfTPaKvipY+rJRB9pLWyWB/dtka+eAzIeTMiGAJ65dWiOFyS8Al0YBFg1xnVaEy6Fl9YNZE9bfdInHbvbSQmd052Du2ri+//KFxS/s+YT+IDt7CZRHta76uc1D9+HanwchT7dQxq2P5tGVE6414xmoafEpAIj8fbX6CyNEF6GQaFOi+77yZ63wOwgmSUTuCF/bRlkjoTHYPYFsoXxqrVA4d7mJ2JNVLAQWrdYycFwHOI4s=' 2023-09-02T13:36:39.860 INFO:scrape:1 jobs: ['7386740'] 2023-09-02T13:36:39.861 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-09-02T13:36:39.861 INFO:scrape: 2023-09-02T13:36:39.862 INFO:scrape:Failure: Host key for server 'smithi072.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHfjna5+y2c8z/g/+Btmghy7JaWrgXmOqW2SBi0PYgqT', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCgcCFxb25c7XA8ZxMeIhkFGuyXxuNZ0ReGJSsrq/tSZB6ylb5ScYDncrOgCG/s7v4Sd2KqrcaWjzpAkgXLaz6bU/GnhRYKdJoqBtjfRH10Om9vreZTnBxmHxtDJVfdbCOEOzWMwNsq3v2G0KyCZDsQgmHz2/oDsPgkxPuSKbn7DN+UB+3ZrV00PRO62Xq3QP41v0/wR83jIgLdKNT2o/NPLJHgtUhDf818SnnP95Ed/E8/k0kGW2EywKnjG8qO4jK89pQih9JSW7+g29a4Dt+xsboNw+sefD373hjB+IC6x5anVUQfF5OpA2QOhr3xcBSWnQq77K38qCBtmD7VprNhq5YPTwX6pPWsiq+w36esK/uzf8J+0C0EmdCUaUZw45gLbxqiHPas3DaO90DmwMIUj72QYCC83cypXANOTCbfhy1XN6yHKWJTL2PdULPYEZMasZIaSGsqGhfzUeNGx6+PifFX9pRUfakeFyniBgon3hKvsL6nBsEp+bszJOpRuI8=' 2023-09-02T13:36:39.862 INFO:scrape:1 jobs: ['7386698'] 2023-09-02T13:36:39.863 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_ffsb', 'traceless/50pc}'] 2023-09-02T13:36:39.863 INFO:scrape: 2023-09-02T13:36:39.863 INFO:scrape:Failure: "2023-09-02T01:25:03.673451+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-09-02T13:36:39.864 INFO:scrape:25 jobs: ['7386718', '7386742', '7386749', '7386777', '7386790', '7386784', '7386716', '7386732', '7386724', '7386739', '7386755', '7386700', '7386788', '7386746', '7386728', '7386786', '7386691', '7386772', '7386759', '7386760', '7386734', '7386723', '7386770', '7386774', '7386745'] 2023-09-02T13:36:39.864 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-02T13:36:39.865 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-iozone}}', '2-logrotate}', 'bluestore-bitmap', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-1-client', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/alternate-pool}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/data-scan}', 'tasks/failover}', 'tasks/libcephfs/{test}}', 'tasks/metrics}', 'tasks/mirror}}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/scrub}', 'tasks/trim-i24137}', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/snaps}', 'tasks/{0-client', 'traceless/50pc}'] 2023-09-02T13:36:39.865 INFO:scrape: 2023-09-02T13:36:39.866 INFO:scrape:Failure: Host key for server 'smithi141.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIL1z7o1wmqyT3ZgCd38P3timJ1j+waW/CzJRFL8/bK2d', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCF0pcjt6Dlks6Eb4dTpNdaCeSy2fnp2O16ENDjXII4kq5gnTY3L5Y8HvYCG7I/XTJAdI89bomkCAqTocubX+c+p0NGA69W9qm46wdzjr7GerLG1BIuALUtkFMDRwZhOkQ2mNq1736NU6N22mjQJoH8Kxulw5VdLJl8J4TJBBQRwV1TeVmyd29/9s/gOagTANSpWbrEYaueafBYdQcWwkwh4xSUewpiPaB13MHB2Eab4LvM2kMxAmosuRMYRD9QGQ4yPs0DETxpNDGkD4mITa4F1VUkigJUYYxgKM/9TpfQ6JK0jkbewG+nkuQWOZ3irhiQUhmEKAQnwCZTh0ERfB5MQ4JFZb/NNWU+mla3jRRzdwZHjwL6bt4R539GGZtAvk/AyMf8Pk7/pkWxnmxMKUxPZO9b8RMiHUC0u7FbcIOBsxJSE3ruJ+SI7vYbSuZYPOeC4Rwx0WD0gAgCVnYZndUXhTG/K4K7Fzv4yirLobo8gZIIHwZG8/v4mU2/4dYhiLc=' 2023-09-02T13:36:39.866 INFO:scrape:1 jobs: ['7386705'] 2023-09-02T13:36:39.867 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-02T13:36:39.867 INFO:scrape: 2023-09-02T13:36:39.867 INFO:scrape:Failure: Host key for server 'smithi077.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPMHYNG1eseoLyP3f2NP8BKIAKkSxgQ7Z/SQuh4oOwuk', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDLgF3XXo7lgtOtQn2ulSeeu+jntzJWYpqw6mZ0DiyLIr49GqZHB8Fe8COVF4npq51pZBI/orf8yecLqZWaJHB2152VGKE0zeMT+dCpCIKcRskWJWnw+SzWDIPDOmSYLvFw+mswnC6MEbWFHdaRc31Bm3wSfdyRCJWc3oWFQjfEhPxzenSse+EULN2m87tWyDupmkK8f8dyZLjYGsaaDFBFKT/xzeImyQ9wnosOVMgZOBxZABW5Onlfvm0AlMxQJ1cx/o/59XJf1s+By05W6uw7zf6bDEqsKVsLSRy8gQT50wlhsnsBpooLlKXjSBKIcZZQRAivak8JZVXfeZXZbqGlIUJGd5HQyQkQvGuTdoVN1CO8mZ2dPnd6ysLPyHUgGrOjzGievwVfYmJfIJhG4pTp8W0OE/Xglv7Oj2kfgczZMwKu4C+0DvDTLw49Q/YmW9dXe+/0Cf84DaM0tAVkQjgkMMc6KRNbp+6EYRNNW9RUa0FDHsmwPDRuOHkb3CUIc88=' 2023-09-02T13:36:39.868 INFO:scrape:1 jobs: ['7386719'] 2023-09-02T13:36:39.868 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-09-02T13:36:39.869 INFO:scrape: 2023-09-02T13:36:39.869 INFO:scrape:Failure: Host key for server 'smithi187.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIu+EkX9Td/A36+3DC6YwFQ/PUBVPfW8g/x0R3WiGgMg', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC47xu4iNjNpGFatkGQPGBBIjBXrw+Oh9DRfpK3rzdDWFfw3FSLwFh3LlkNqcoBpW2+g/XKd6mP69GWvLbaOZVE3tGP52ClttnfMWhScy42k/pWWXpP4gZSWhRr1RjArx3Nr3uJfUQz+3wMevtkz1sMl4MGjM+Yryx3RjtALbbCdKxM1tCpYf8qr528u7X2rER8OQeTZvWsPIXmbOOgh0IGMy1wvzDkyITMiouAAtBaQgwh3FaBk1hO72/Cd2vvSEDA6H0x7smq4LBMdyl1Th7aOyqEkLQh/6nJ+TejCC/yGYEYWjco4W6kIBRzMOzvLHpZ4p36b+Qbc6ZExArokaMiBRc0i4JDyLIN1FCrLYekqeti/0u7ls3dXcrsbcYVLnNgzNk2GbDR0JL4axTpVwHX4qp/310h/Scs6Lrm0eOSCFWO0BUs8Zh1smSUe5fLGlJd0N/TieqNZBFFOqrDFDpHJm+VVINT/AZwvKhFOu0fK7A2ljXtadDCXko3tHU0+RE=' 2023-09-02T13:36:39.870 INFO:scrape:1 jobs: ['7386741'] 2023-09-02T13:36:39.870 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cluster/{1-node}', 'fs/top/{begin/{0-install', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'supported-random-distros$/{ubuntu_latest}', 'tasks/fstop}'] 2023-09-02T13:36:39.870 INFO:scrape: 2023-09-02T13:36:39.871 INFO:scrape:Failure: Host key for server 'smithi149.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINIdCgHQ2nH3lZl9ZbfCuhSotDyNNW7ra0i6G1jb0I25', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7ISI9fnHl0DohWhcEz3dYUZxAvKV/hUj5k/YhasC02kWJEuLqcks1zFyHxmI3i1Ij/ahpevREmdY6pXlsFe7AcBr8ctlZL39TeF4t9ETv6plCUN4VL+amqLHazDV8t4HNw1e/cUoctoNHtwWZ4P2PUvx545DACpcvMzq36qQwQkLEIX2wUrngeXelA3GfzTJrmzYD7YSzXHFFajaWbyYMMCrie2yr7gSTIFLpGp5nTdAqZyRJSAkql0FgXiTULKLF9apWOuIGSddDlCPje+UEfANBtwgXmZJI8hg1vQMn9ivzfjQPWG+Z/lVIjtLz73LEgO/jSQJW6wpxfalNJJWra0SgnHISJZozN/U0ZI5zZk6eQMPS96zEafGtU5gf49h020fXxPtDfgtBva6ouLaKDn7JxQfjEChx3i514UrzELLUCfiZxnC0De7LKHK9+oLny3/fZDNiaiT00kPY7SflEeO4YTumltIb4wD0vHt5t8CWsZ0rbDGO/zbf+7dmM=' 2023-09-02T13:36:39.871 INFO:scrape:1 jobs: ['7386783'] 2023-09-02T13:36:39.871 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-09-02T13:36:39.872 INFO:scrape: 2023-09-02T13:36:39.872 INFO:scrape:Failure: Host key for server 'smithi114.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK5p6LrQe30wfff0x8diOanKdjYqjcVGEW2LQyLTohNK', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDiTc4rcTiS+e9Ie0cWOQm0W8ap+L+GTGj4hWnfIOY9SvIJbIzKeoFSPFLxIpJ1iWxRuaPa+wzUi/7M5ClwC/Ehe76xzQhEARRcK+84Ws5+CDdzDHL/OynFrAonkiTqIFEsGZB0Je0CIbmc8BRSADRBYBWMuzVceRbQl4g8yjdj9NN9uX5M5GmX3BuzqCki5mVe7YbkntN8iSQHY25lnZ/QwQpLWb1G9XUMv0dt38upoATRe/zStvMdrUCuySFO5XhkG3H2upclIbPDM2FYIBs3x8aUil9VYzi5Jyl9Y4uBAjKFgCdJSTuihgtkSEUops/7xTZ1+V55OZ9oIf0C1d1V99vZ1q88/0FMRIBPuAnOQbx6BowBbaquPJGz6VkYRDfCqTHbeQA+OtfFDQM2R+5QCm3klKw/sOt974CpjYDfN9o8ivUQn4p9efqWUCVhFTwyY71kK/vzEHx7wr9M6re414NqvRW22AiaZ7hiXRHfNLPkLW/RQ2Zoe+vx7Xxclfk=' 2023-09-02T13:36:39.873 INFO:scrape:1 jobs: ['7386787'] 2023-09-02T13:36:39.873 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-02T13:36:39.873 INFO:scrape: 2023-09-02T13:36:39.874 INFO:scrape:Failure: Command failed on smithi078 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" 2023-09-02T13:36:39.874 INFO:scrape:15 jobs: ['7386771', '7386729', '7386708', '7386722', '7386756', '7386743', '7386780', '7386714', '7386762', '7386791', '7386748', '7386782', '7386699', '7386753', '7386696'] 2023-09-02T13:36:39.874 INFO:scrape:suites intersection: ['centos_8', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-09-02T13:36:39.875 INFO:scrape:suites union: ['1-client', '1-mount/mount/fuse', '1-tests/dbench}}}', '1-tests/fsstress}}}', '1-tests/iozone}}}', '2-upgrade', '2-workload/stress_tests/{0-client-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_8', 'clusters/1-mds-2-client-micro', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus', 'tasks/{0-octopus'] 2023-09-02T13:36:39.875 INFO:scrape: 2023-09-02T13:36:39.876 INFO:scrape:Failure: Host key for server 'smithi110.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIBs3fimkTYa6hxggLrpNA5PsopQlseyRtaWUvl9qOE/s', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcsTNnRHZrk388jboeM/fF7p3AXlOiYYoC21Y8X14RBVdsueYLKhN0QA7QGp4BBBP6JXB4eDO78BRwSo6ap6L4bgIPEfD4jTUdMNJ2vxPyd/RmS2NzxPnEryRwDIUS4V5EPzvfqHF+SpBYhrTv9AB2MuVpQTWNAMAkmEoWCWenaQHSoRvGQFgWjFY4IaC6U1FMQ9eLRuAut3PMnxbJxT1iUKb9qw09+C0tlEtZEaejMikIZVLQDVmRe6p0BQ+bHpoXJ+a/BghWEXXYBVgkfN+ot61EDN3evYbIxJH/yuejf4SZC3tXUJmCAaAASiR75cWXQGBnvmijS5DQRY6o81fxxeiF2WIqh+S1E0wJ1IH6Q26CgOuPrJKgYvrZyAR4mFK84o3Ej0iM4VLt7CeX2zXvltw499/rUQu3WM74XbJpv0U/aHbR/F6dEG4jxihTiifwF7+wBCnbPKAYBMBbb4c5hra/2CboZDfp8hS78DMCTv3KS1VWfJgrn+DxamfzpM=' 2023-09-02T13:36:39.876 INFO:scrape:1 jobs: ['7386735'] 2023-09-02T13:36:39.876 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-09-02T13:36:39.877 INFO:scrape: 2023-09-02T13:36:39.877 INFO:scrape:Dead: None 2023-09-02T13:36:39.877 INFO:scrape:1 jobs: ['7386769'] 2023-09-02T13:36:39.878 INFO:scrape: 2023-09-02T13:36:39.878 INFO:scrape:Dead: b'2023-09-02T13:11:57.555 DEBUG:teuthology.exit:Finished running handlers' 2023-09-02T13:36:39.879 INFO:scrape:ceph version 18.0.0-5814-gcc14fc8a (cc14fc8aa878d09ac353379f17aec3b92248b995) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f1d5ee54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f1d5eea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1d5f7631bd] 6: /usr/lib64/ceph/libceph-common.so.2(+0x163321) [0x7f1d5f763321] 7: (CInode::auth_unpin(void*)+0x266) [0x558d1088cff6] 8: (MutationImpl::drop_local_auth_pins()+0x39) [0x558d107262e9] 9: (MDCache::request_cleanup(boost::intrusive_ptr&)+0xe6) [0x558d1079ee66] 10: (Server::journal_close_session(Session*, int, Context*)+0x5b0) [0x558d106b8c50] 11: (Server::handle_client_session(boost::intrusive_ptr const&)+0x1932) [0x558d106b3632] 12: (Server::dispatch(boost::intrusive_ptr const&)+0xf8) [0x558d106b4548] 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x558d106396ab] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558d10639b9c] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x558d10623265] 16: (DispatchQueue::entry()+0x53a) [0x7f1d5f95b5ba] 17: /usr/lib64/ceph/libceph-common.so.2(+0x3eb6d1) [0x7f1d5f9eb6d1] 18: /lib64/libc.so.6(+0x9f802) [0x7f1d5ee9f802] 19: /lib64/libc.so.6(+0x3f450) [0x7f1d5ee3f450] 2023-09-02T13:36:39.879 INFO:scrape:1 jobs: ['7386704'] 2023-09-02T13:36:39.879 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-09-02T13:36:39.880 INFO:scrape: 2023-09-02T13:36:39.880 INFO:scrape:Failure: Host key for server 'smithi093.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJ5SpGprLkXEA4auuXrNcnMBac44pPGe4erf2G6Uwhcu', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCyWN5NG66e3qSeb2+/2lGsi6FR+CJkXb+M5878MAaf/4juQUYRmOw6KCztWtgqxQXWPVOHa5j3iye+kTE+UfrguBhYaeqk1ZWpDfjIl5aHuJrbd7EISz3E8pAsuJQR5JCqruIWjygpWZyOeschjwptrzqPh65NrI1gNe9EDSUbJ+t+9GQRBt6vm6iSGs5V6i6XWRd5rBsfB2gQLf6hVr7nHAN/hJlwa6eikkm9hYn0TIYnMGQn8ybQcTPSe52m5GQzGgcgr2IzMZXf2puUWl4QzfFwm+awmhMxM4tnPuLHhs0RnBx+nIQxWSmjWvEr7I0o2PLdlN8FKuRKdmFPBzEWTVbH2MYhzCVyihgBpWVjLfJWOS7vA9OHCXS4BoyKSpqRgUvobLKObgHsMJCrpju/WU/fWgQ6ZbefUR3FJQd9K0lMfWrQ0/mFGvGMmrhBaDJZ46q9SmU46hBuAajOHQn+Nab1PzWPB29lhP6duwYOoKQi/TOp/KuKEJV1vJqCpDk=' 2023-09-02T13:36:39.880 INFO:scrape:1 jobs: ['7386693'] 2023-09-02T13:36:39.881 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-09-02T13:36:39.881 INFO:scrape: 2023-09-02T13:36:39.881 INFO:scrape:Failure: Host key for server 'smithi121.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIN16Ld9sZ1KqNlrOpDGrk7Lkx4pe0eIMTVrgcguDmQau', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxnC8HTx2f+WtGzPoV5FFdJMelgAMhvDf8sabyGCSJpWkteb9CKgUIpqvjEYrcaiiJ44xjCCNL51wW1XkvxKKcRDfKeK8Z6dvlMZbvU6BTfWqUDFIfj8Ufm/uWm3FIs5cbXzB5M6YQYALicHaNITAU9XnNX+ootgdb2OY8RXH6MUfq5cOAxrT9HBbjPYXW/SJUS1v8Etq6YSNjLrH8XOIiBjyL8M0pjMyb5UFfgf82m1BB0puxRLONhZB9uNMVcbHiP64WV7uwMiIrGSwtgLxNtww54KgfJL34d1wzXe91PqlYkiY65lrkLalpK4VvfjKzACqqEVbVLy6/3wT/E68XqnjnU6mfiJ4XV32waD44mfzmaLwnk0JcIz7caO3ertSvkIPaNSXX8qbuMSWffexHlxEarYcN/gedhxQNcYF2nMCyAMuxvDS/pKNCNN+FliXzkXFThaJYFJRfZ65XcFFOLnokCeEkJ0E0Nis6U1XHL4q4bRjaigrZAu5k95Tmsc=' 2023-09-02T13:36:39.882 INFO:scrape:1 jobs: ['7386767'] 2023-09-02T13:36:39.882 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-02T13:36:39.882 INFO:scrape: 2023-09-02T13:36:39.883 INFO:scrape:Failure: Host key for server 'smithi119.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIB4mkPz6WW7wmyPyep1QmctB5/QRuydQ5rfxL5DiyV6g', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCcY/ZMXh1sm488nIXOJq+3dBWeDvSeZ14C9bSmSMyOCDAX29qFMz+2m9D97j8bGlz4brif5CJLPtLA+vBK6/yzhffZJb4oT1Y+Vo06gKURdOYDYyQA48aQ98dHM1isas73oolyguyQdD+Irtrak1HyUvIlzw9zplT74pOJwuPrsWB91ksaV4WNAJMKK8rvTzv2RjuNV2Yh0LYePV0jNLi4XeChF4pTxaKD3SqXsJKlmK6UrVT4G3vrZa4sd+0iqhJKsGEFSg858aiFmSrzhmiJkXhtgeAnGTKoVZVrIYpJXjdJkfBoHwune2MXtCd69RFIbnIYlC8IA4jzEMS10T6QQayS7jf7c42manMyVSuw3okrdDtKvm5z1/y9DkQF3tGTXL2AgJYE0r5qvNEXx/aYJDJwNPgRWN66MS5x83aMu640rmzTmvf/fItSxdyoeJn4mEn3d85XyVrppBKxQL28RlJ4zPORyKW76jsmgSfTClJ5mqtO8D0p1igUY6Lu2Mk=' 2023-09-02T13:36:39.883 INFO:scrape:1 jobs: ['7386763'] 2023-09-02T13:36:39.883 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-02T13:36:39.884 INFO:scrape: 2023-09-02T13:36:39.884 INFO:scrape:Failure: Host key for server 'smithi121.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIC7lIic8ZLn28Eu79yvAjlVrY5egNkr39QTuwRxYMphH', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCsKeH+pfAgQseJeYALNyr2ZUrUZYF5bSsyhVbYpHMgBHNHtyCv22M6+pOBtfwBEpBOOfxS4GuEJGatSFGNmC07YuCuQIRD85zRzXad7m9JCJVcj6JBiWwUl5QDOOmqocR1drWTiGEilsx0F3wFnj8jupxlk3R3H+w6hmzsnvEFKRHEpJGho2px/kRxIeGFHpcs96bmQ8ymhBqVDCTwVTxUpfBawyeWJyt24ecRX1GyjwOPcDNwUDuORMkgHWTpiKAXKbMJBdDJ1uiKfhef2Mj9gA255KbOt0Eym2bX8krU+d2FiTdZto+7IqtRsrY6I3FnN2t5PdWCmJcl/QRDmcCXGvXscj3AgdqySICzCiEzk60a4NPt7uELc2mmKNJVwZADcIsAQonovgGblD/0JshKVJBTuJ4sVLbif+Ee/usLmYFHTla+T48Ja3uvTYpIR3JsiCXVOq9q/dyd7u0m9co1sb3NN4Ynhvft2I7lRPjIXRLEeH9DKL3EJ7S/aNS9oBs=' 2023-09-02T13:36:39.885 INFO:scrape:1 jobs: ['7386751'] 2023-09-02T13:36:39.885 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-02T13:36:39.885 INFO:scrape: 2023-09-02T13:36:39.886 INFO:scrape:Failure: Host key for server 'smithi173.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJ2QgIwzcZ9y5sisDbqg8o8/xnSxz13wof56Prs3QeEn', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr5QSOhkX++BSdoGfZtWvcn2YTacWQ3sfpRf6mL7WKm+g6JepXcWIdbFhmEQ0Wutmb3PFxFSG2qtqUrg8T3aJGByOJRmHTgap/tX1Zl2PsBsIpwknqUTXge7JIoj0OwJ02qf4BNGu3I6SWU9ZK3aTOiCTkQGCzSeOVn5y0RqzJma5jUKW2rvpzSFDuBLWrAeMnqWTjLrV9wF4OQi0xL1x0mqwRbSgP34qWF7Ei2s7s4RH/vu/1RZnV7fiy6+LwBjlF17rRnii3apYtipFE09XJlw+AlQ15GBnDDQMEYDASoDyBOcSza/k0J4AbeHOiJuXrmvVdX2OD/2XFCFIDkmUFXX1B7uy88UqgzOVLx+0r00hT+d3909HMakwSMvIjPVTAt/BWfbsWM39EZVPK7M7waLijtsfpesM9j26BOeQfZjrwExe5PPu3WB6uo2j03M6La0POW9HIrDkqey4UD0l+nn6ZyqHMbFroymX4CZ6Uf91VpHdQvkiUuXQj3z+trFc=' 2023-09-02T13:36:39.886 INFO:scrape:1 jobs: ['7386702'] 2023-09-02T13:36:39.886 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-02T13:36:39.887 INFO:scrape: 2023-09-02T13:36:39.887 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-09-02T13:36:39.887 INFO:scrape:2 jobs: ['7386695', '7386779'] 2023-09-02T13:36:39.888 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-09-02T13:36:39.888 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-09-02T13:36:39.889 INFO:scrape: 2023-09-02T13:36:39.889 INFO:scrape:Dead: b'2023-09-02T13:12:41.782 DEBUG:teuthology.exit:Finished running handlers' 2023-09-02T13:36:39.889 INFO:scrape:4 jobs: ['7386738', '7386768', '7386706', '7386721'] 2023-09-02T13:36:39.890 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-09-02T13:36:39.890 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}', 'tasks/xfstests-dev}'] 2023-09-02T13:36:39.890 INFO:scrape: 2023-09-02T13:36:39.891 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-09-02T13:36:39.891 INFO:scrape:2 jobs: ['7386781', '7386776'] 2023-09-02T13:36:39.891 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-02T13:36:39.892 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-02T13:36:39.892 INFO:scrape: 2023-09-02T13:36:39.892 INFO:scrape:Failure: Host key for server 'smithi149.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDd6duO11mRVyqhuzhxlpV+PTlHsf8mDTFEH40EmZ15+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC1PMUnG76RsAnlEz5oatc0uGC5Pw+3KrnPFp3PnRIhz6t2OYfzHltrwRgr7kawR/H0417pzi91vxReguaLYB9nsMUqSVTHICan9Sb74J//CXjvlGuy+z1v9jftPds5xsb/JBnKZR65xNynM4Pm4MAH0Z9eXmNB0N7SYkkGcbY4nbIt8SytZHcmwxURfjGAWNnsd+DxgR66q5Yfyh3h4R5yI45VgpP2A6fcUGCD8h4jU55rysgMJxyIvt2dEfrNrOuNv4AjAL4uf8N8SX7lcDXwysu2+KuGy6AwFwLJXruxolzavlZga2hdPsDYxzcrm6KW6X2vHDdjWpbihkDRzVVqwpUx5gNqRVv1WcXrxMixQ6MAXZNIa2Wydp/50jgv7K0cu1giFY61j4IHZZ+Srvy6HNi7UEZzg317RSJpQ/u/ctP7xEfF02vI7JW4kl4SCSupMjMWskgG4T2pB0qs0r6/o09IpsvXnzerGkMIuBKjAokmyD/bQbEyQ/d2s6fFWfE=' 2023-09-02T13:36:39.893 INFO:scrape:1 jobs: ['7386715'] 2023-09-02T13:36:39.893 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_fsstress', 'traceless/50pc}'] 2023-09-02T13:36:39.894 INFO:scrape: 2023-09-02T13:36:39.894 INFO:scrape:Failure: Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPd/VI4W+lOLvjJmfW0bBHjSKVEWJXhbZyxBR00rgUwL', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCfr6fuG41jS6TYteu9WpDiK7ulUs3gib2JrJqSVwDXcp0N43qLRJprGK+sdVcwUq98Tg3aDzFTCT3+P70y0xMfT0CHc+75DuSBedhjOc5/mG3lgZPAHTztdU5nbyZbtArMkxXemE1kWMGbp5GPUPV8CHjnftBvzCDjcB2Z41Lsc4E5tITEde91wHUrEhJX6CljkT3PNYirfiZYSgnT3buY+7KvBsw5ZeX+3ROxo3nZNvuKl8o2CmDuvketi/9JGCId3JvsBJXZMUbba61XMOmBc1x85YlXQMrCJzJcVmOXhFxbUepFXvN34xTlMh0fP5EHwFSzhB/mKLRNRHMLfZHDiTW3+mN8h4qdJmEFDMNhPvF77MPl0OJ/zXO/PjttF9VtJH3VCeMu79Ra5lKLEazUBM+QJ8aWQLBtTUk4QzVQy8X7VUJ31zuWyPEFgtcCNGChym3/8Nwh0YDUrQrW6BI6Q4o2tbT99WPWniUNuwhmKUvlUrYHCz/o/PmnkB28rZM=' 2023-09-02T13:36:39.894 INFO:scrape:1 jobs: ['7386717'] 2023-09-02T13:36:39.895 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-02T13:36:39.895 INFO:scrape: 2023-09-02T13:36:39.895 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-09-02T13:36:39.896 INFO:scrape:1 jobs: ['7386725'] 2023-09-02T13:36:39.896 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-09-02T13:36:39.896 INFO:scrape: 2023-09-02T13:36:39.897 INFO:scrape:Failure: Host key for server 'smithi152.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAID4/dYjNimwXoGxr5J0uSU3pfcTHf9kClNusASo+fzgA', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCW2xf8sc/z9FeroTpjtq1uFyfhzXb/EkPjLV3qzdHov5zPlzpHano9xd2XSSeGoBHDfeTJ6QWIS0WBohzWuILCGIuMVWN66z94tL3FlUHoAW6t51EE8KBBJD4V/oqWISEm7Ej0i/WspNJ/kXW3nfhgDY49TcZmP7nNSre5f+aRiVU7Bbkbl75thoPqcPJrsFv3Hc4qaqeuq3I6fshGfC+ppjV0shARCfOuAISib2YSJkLtg4onohKQSmUU0zQbnTo1BFAalKEOg+s5EXCIBBO8tdWMiEPoFL4ra5P0gN43fGZ9z0IBj9jV2pN3niM9O4zDO0Wet0Ofyi9xwpKLovoaDZbDdLdMfXuVI2uZ3tOuAeEvsRvc0twTzytIEvHGcftn6JhPMcUb5Ye9RT7l5ROPpgPP8aWAx7dsof8UTKSARCy/7lLJdJL+ls8R5Ni05CGVRzZUUdUFtIHbGKTXeUghZXXn5xHRvYMW7aGsPS59X9lAE8hTXnRofIkaX7o353E=' 2023-09-02T13:36:39.897 INFO:scrape:1 jobs: ['7386736'] 2023-09-02T13:36:39.897 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}'] 2023-09-02T13:36:39.898 INFO:scrape: 2023-09-02T13:36:39.898 INFO:scrape:Failure: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK1KyS3zIEFHbrpbjdQQy8eGUdMJ06f7lfyv43wkt2Fb', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCURBCz4YaXieURDkEHkLUIngSDnCFQjnniF2D+NNeyxDKTsGQk5UUqnsJbKGe1ikRhC8sAe6b9hTZCLcx1+pVgI1ZWSoYFzBiJ0rSjLvhJEji7kjnmdeVjvhWhiLofAmbJZFL9OcBYvt86lvkFukLYjJ1w5BCVWHNodqj3xULpZnfyUw67312CHcAlfpLQ14gRRjbFcDUTeuI/ZZMOd3rZCBB0j5oT5W9ECYNfWaeD0W1ssHQ9+XcYJt8flB/m+zRrvq4106yr5aRrgLhM2qk6eSPJWlqbxPWFlPT8yLffTMJu7SdNBr+WvAqGi4hG1Rm6s7dT1HDBQ/hvwYafqd7abKpfTgsbtqIffa5NdkhMDx5dGynfA8MXLxPOqXWKOyC83FAoTl097U2gJ3CcyJddtP9+F6dakzUgMeG4bYfEfbzDHEzYTY+kOxgEvAw1vq0JQ07jgXWK+UUbysmRHbvnOV1r7qtG/FoxqfbWLPewhHFm+AE0icz9iQncRnK/aRs=' 2023-09-02T13:36:39.899 INFO:scrape:1 jobs: ['7386757'] 2023-09-02T13:36:39.899 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-09-02T13:36:39.899 INFO:scrape: 2023-09-02T13:36:39.900 INFO:scrape:Failure: Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) 2023-09-02T13:36:39.900 INFO:scrape:1 jobs: ['7386754'] 2023-09-02T13:36:39.900 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] 2023-09-02T13:36:39.901 INFO:scrape: 2023-09-02T13:36:39.901 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2023-09-02T13:36:39.901 INFO:scrape:2 jobs: ['7386710', '7386789'] 2023-09-02T13:36:39.902 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'tasks/cfuse_workunit_suites_pjd}'] 2023-09-02T13:36:39.902 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'fs/permission/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}'] 2023-09-02T13:36:39.902 INFO:scrape: 2023-09-02T13:36:39.903 INFO:scrape:Failure: Host key for server 'smithi046.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAID+OQ5CVMJL3iHHEAa/jxQAgt/2KdYYzYy2KkAPAQhW0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDv63Rn8WNUo8iI27vteuDXMryU4QH9ckGcPDFhTn00peQO0MbHwpF9IE+HySKsKTNUXpPfsZOaoGASfSTIUuaQKMbpVE8SbkBNSRqjp+HHrusGVInnFfdLalnDp50xPxbKWnuYuM+Rq/6UvEP+dvwEC8XoO2TZSvBp+lIsoWwoZWcpm5qZU4IidXLCY4kUf0BSly+qt2eQCBTH79UlmaOvG5ANnvA4OTCPvGQN1MZGJon7V3MFI0p4s3pSpZYes95GFWApLETLFca4KfGfNGMQ5SKt+2eqU2O9hBpwZzYt/aRhyDZrk6HjcKTPWi4tfRgRbbR+iRWGB354ZWRBo/adW9NKVuAnjvQKS0lcb0636WzH3EhIHg6aLvRjJy62d5YzjM0r23g4YOy7vU/6UNj7oHlDgpdjl/8mrbwPJmwg0LZh7Xwsyb1j9ivTNUZ7Rozj3PF7ETSud6wEYCFxx2RoB8Pwnm5EXZwZbw4amFKyXDGY35rWUjKX5lb2Rv9BObM=' 2023-09-02T13:36:39.903 INFO:scrape:1 jobs: ['7386737'] 2023-09-02T13:36:39.904 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-02T13:36:39.904 INFO:scrape: 2023-09-02T13:36:39.904 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-02T13:36:39.905 INFO:scrape:2 jobs: ['7386713', '7386750'] 2023-09-02T13:36:39.905 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-02T13:36:39.905 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-02T13:36:39.906 INFO:scrape: 2023-09-02T13:36:39.906 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi170 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-09-02T13:36:39.906 INFO:scrape:1 jobs: ['7386731'] 2023-09-02T13:36:39.907 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-09-02T13:36:39.907 INFO:scrape: 2023-09-02T13:36:39.908 INFO:scrape:Failure: Host key for server 'smithi183.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPd3z/6y2UmX8a/bgkTLh9vRureYBg2xL6FqdZfDaj27', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCdhzxUo3cXBZyYSVBmhDfRqZsH7OmF+k7JA9qjkPRxbU+Y91JtKEMvp8T6TEurnGNh+x4gocKEvod4xwobMvw8vhyuqOrGDl0ZfHiAjQDGjkys9exp5s0y4wNCAPtnSlyGDGXDIL+Jge2IHUb4p7UIdCsHgLdPOofubTxYhgNdG/RoFGxe6j8UbppZkUtaF/sbuRa7c3/oW8O39CpP1HsKUjzvQiE5KlcDcT697ev0FLUsPqGXNflktf8EHvU+3QgDUNCefW6IoOSJG2IeGTZRtPvlG/LxtUOBUg1r2Nii2v3qgpgaR8FbgPUTnEljiqgR23cGP5CZAYs3NIs0C0qXWOXcehsbqg5rgJBH/6Mx7GiTWa8WMyt5sr8wuyda7e+kxDdOsIPFLC/gKLMwrLchOa09jIgaF1bAeFQq3Jp+jYGu8TptbmdhmbllAeK2wVIXA/dGGloqMagYMW+ToGlz6IKEpsW/XOQIwc8ZIdoawVUOBG3taBDzd6jV/oSwBA8=' 2023-09-02T13:36:39.908 INFO:scrape:1 jobs: ['7386712'] 2023-09-02T13:36:39.908 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-09-02T13:36:39.909 INFO:scrape: 2023-09-02T13:36:39.909 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-09-02T13:36:39.909 INFO:scrape:1 jobs: ['7386711'] 2023-09-02T13:36:39.910 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-09-02T13:36:39.910 INFO:scrape: 2023-09-02T13:36:39.910 INFO:scrape:Failure: Host key for server 'smithi018.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHC8w76tIocNhdPexjs2ANzNo7GtzsO43y0c6avVY/dQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCg6XY8VbOOuhPvS0GtXX8P6lQhaImOFxrjL3MFvZy7dxXXKbZGoMKF10vh2FtK1e6NUXnwGuNI6GdoZdKrDRFAShBV9tKM50RDG2v2hFM3q2GxxM4BLD5Rb3uvdL6oxO15SIoNRNZPH5cZsKt1/kpawhZa1WEW/0A/EJtGQLRy0neUJAy2B8LQ9nssEXJRUy6SazmjDls9bADMhCNDxZFJlZXa2KJcmVPsVv4wkq/Pzf1S9NyzI9t+k9Bgr9lLbUdHhddTQTkA/vLIrb7Vfouyl+i85IxWx3kyEDRLGa5EwYc3q6hEZ1A7LLf29+b6KFawS2f20yCEUYqnqWxxRWYp+gmp6m0+BLh7y0q1W5smIWckRpKw3E1pqGbBdyOwcOWa7Y2FxtnyKyV2Ylx+SSPHTNJsSamsesVJsM2pLVY5+TV4P+u769C4dcq84uKbfH3RvCJJc3ET1dqniLsEZZFQ0i08VEyZkpPMJ917TYABlGuZSu3QAPpPLw7Z/n5/jyE=' 2023-09-02T13:36:39.911 INFO:scrape:1 jobs: ['7386778'] 2023-09-02T13:36:39.911 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{client-shutdown', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-02T13:36:39.912 INFO:scrape: 2023-09-02T13:36:39.912 INFO:scrape:Failure: Host key for server 'smithi155.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICIqMpbUL3JGF4AHPexGGQnBxFvGIA+PaOLyflaBSrgs', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3IVVQraQRAvogAPiZq3+iwMptfsPaUoNiLyW/73d9sqxDJgAeW0yVfU2phOCHGrG8wDIYUrs6FMmvT0cvYeFsR/hgups1bHYeE4w61Xwqs7Iox+AZEYdOzT5o06QGYhgBLiQvEbp5Iko4/aW18iQqXdwgaoX3NJDjZclcfEEduAg76vG0RqlLnHG7osP3EMTC9hnZYgRpAVEER88cghuSGO+jDYCI0mkkuY+2iecWcVkVDllvxVwVa+WUm3FSRHxArL2bFh1l8VsjduH12w/BG7QX1t93TdmmU0iq4PFVWlyegJ9B9M8R0ogV29HG3AszEkRsWQNvVrAFZwJ+pOxp6av3A3pZpkGqN8oKiAuQ9Ee45o6AM9tZozTlzk6lGm8sOtV8MM1st2xMcufH0hlWZGLk0TtmWVBQQ5d77T/lVtIcF4W3PV7bDNIaCIgf8BJfo14M00ZzcPtrf418lgLzyI0Ylp6BbxLoE8liU/Au/uiB5321JLzxdPCivjHFXkk=' 2023-09-02T13:36:39.912 INFO:scrape:1 jobs: ['7386764'] 2023-09-02T13:36:39.913 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-09-02T13:36:39.913 INFO:scrape: 2023-09-02T13:36:39.918 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/ failed: 75 dead: 6 running: 0 waiting: 0 queued: 0 passed: 20 Fail ================================================================= [7386691] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386691/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386691/ "2023-09-02T01:25:15.652527+0000 mon.a (mon.0) 252 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386693] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386693/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386693/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c52fe690a48b4b2fa2950d90c74dd8ad Host key for server 'smithi093.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJ5SpGprLkXEA4auuXrNcnMBac44pPGe4erf2G6Uwhcu', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCyWN5NG66e3qSeb2+/2lGsi6FR+CJkXb+M5 878MAaf/4juQUYRmOw6KCztWtgqxQXWPVOHa5j3iye+kTE+UfrguBhYaeqk1ZWpDfjIl5aHuJrb d7EISz3E8pAsuJQR5JCqruIWjygpWZyOeschjwptrzqPh65NrI1gNe9EDSUbJ+t+9GQRBt6vm6i SGs5V6i6XWRd5rBsfB2gQLf6hVr7nHAN/hJlwa6eikkm9hYn0TIYnMGQn8ybQcTPSe52m5GQzGg cgr2IzMZXf2puUWl4QzfFwm+awmhMxM4tnPuLHhs0RnBx+nIQxWSmjWvEr7I0o2PLdlN8FKuRKd mFPBzEWTVbH2MYhzCVyihgBpWVjLfJWOS7vA9OHCXS4BoyKSpqRgUvobLKObgHsMJCrpju/WU/f WgQ6ZbefUR3FJQd9K0lMfWrQ0/mFGvGMmrhBaDJZ46q9SmU46hBuAajOHQn+Nab1PzWPB29lhP6 duwYOoKQi/TOp/KuKEJV1vJqCpDk=' [7386695] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:15:04 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386695/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95f0a1a82fd54813ad3dc91a8d665d4c Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7386696] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} ----------------------------------------------------------------- time: 00:09:56 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386696/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2870d4352fa41c8928be99006d6dca3 Command failed on smithi124 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386698] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386698/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386698/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d3689951370e42c89f475c2de114d7c7 Host key for server 'smithi072.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHfjna5+y2c8z/g/+Btmghy7JaWrgXmOqW2SBi0PYgqT', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCgcCFxb25c7XA8ZxMeIhkFGuyXxuNZ0ReGJ Ssrq/tSZB6ylb5ScYDncrOgCG/s7v4Sd2KqrcaWjzpAkgXLaz6bU/GnhRYKdJoqBtjfRH10Om9v reZTnBxmHxtDJVfdbCOEOzWMwNsq3v2G0KyCZDsQgmHz2/oDsPgkxPuSKbn7DN+UB+3ZrV00PRO 62Xq3QP41v0/wR83jIgLdKNT2o/NPLJHgtUhDf818SnnP95Ed/E8/k0kGW2EywKnjG8qO4jK89p Qih9JSW7+g29a4Dt+xsboNw+sefD373hjB+IC6x5anVUQfF5OpA2QOhr3xcBSWnQq77K38qCBtm D7VprNhq5YPTwX6pPWsiq+w36esK/uzf8J+0C0EmdCUaUZw45gLbxqiHPas3DaO90DmwMIUj72Q YCC83cypXANOTCbfhy1XN6yHKWJTL2PdULPYEZMasZIaSGsqGhfzUeNGx6+PifFX9pRUfakeFyn iBgon3hKvsL6nBsEp+bszJOpRuI8=' [7386699] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386699/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e73da8b1602467088cf2359abe2818d Command failed on smithi042 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386700] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:10:28 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386700/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386700/ "2023-09-02T01:16:51.445844+0000 mon.a (mon.0) 186 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386702] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386702/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0ea375a1cc2481ea820c49e241cac57 Host key for server 'smithi173.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJ2QgIwzcZ9y5sisDbqg8o8/xnSxz13wof56Prs3QeEn', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr5QSOhkX++BSdoGfZtWvcn2YTacWQ3sfpR f6mL7WKm+g6JepXcWIdbFhmEQ0Wutmb3PFxFSG2qtqUrg8T3aJGByOJRmHTgap/tX1Zl2PsBsIp wknqUTXge7JIoj0OwJ02qf4BNGu3I6SWU9ZK3aTOiCTkQGCzSeOVn5y0RqzJma5jUKW2rvpzSFD uBLWrAeMnqWTjLrV9wF4OQi0xL1x0mqwRbSgP34qWF7Ei2s7s4RH/vu/1RZnV7fiy6+LwBjlF17 rRnii3apYtipFE09XJlw+AlQ15GBnDDQMEYDASoDyBOcSza/k0J4AbeHOiJuXrmvVdX2OD/2XFC FIDkmUFXX1B7uy88UqgzOVLx+0r00hT+d3909HMakwSMvIjPVTAt/BWfbsWM39EZVPK7M7waLij tsfpesM9j26BOeQfZjrwExe5PPu3WB6uo2j03M6La0POW9HIrDkqey4UD0l+nn6ZyqHMbFroymX 4CZ6Uf91VpHdQvkiUuXQj3z+trFc=' [7386703] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386703/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386703/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52dbdbfff0a7425aa4b0a5fceb27e494 Host key for server 'smithi097.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIImUrwqDLKVvSGsEWfds752ApEoo4UucmgKJ+jzAhKtX', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQD0bdzCebIPvnj2DJYPXZG7oeitjm3x1jJXR 4IifxxY9TqvHX6bc+Ao8x2kd39cif9jOJFHxCe24qewXEELaqD29r2hKRNSASCZB77ne8lDsfNg MZ4oaKd9pQr3h5bNqTXp7Tm92cilhFWfkavEZVGLAW0BsdjqQJlj5/fQ0jaIrJYZCpqdqQI2WcG FI2NeHSbmSK39pR5y7nxTMiOZmywAvFXMDEf8Ywuh/jjvtl0l//8QK8tXWpaITtvseLBLaDkvw5 VICF1C+5x2wkJmxFEAXAm3xyz3vVGXu4BtHh8EeMF3p+2b0IzTNuGlhX6jKSEjLSvenqDkHzKsa zwX3vcl4pAucmoztk30Fibx+rT5yjX/hcljnf9y4kQxYrurh7QO/mwH9yuG62gsCwPL8uF4CQJl k4HcwYuGpchwQfAIZZz0jsACF9R2Nd0s995u8d/wlCAqFPCp4pCH99kURMmXX/nwJWHgGqTnh3W GA9MTe3mouGdHs9Dvw+Ju6K5F2SE=' [7386705] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386705/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a24f8c3b16f4398adf5446f1f1b1c7a Host key for server 'smithi141.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIL1z7o1wmqyT3ZgCd38P3timJ1j+waW/CzJRFL8/bK2d', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCF0pcjt6Dlks6Eb4dTpNdaCeSy2fnp2O16E NDjXII4kq5gnTY3L5Y8HvYCG7I/XTJAdI89bomkCAqTocubX+c+p0NGA69W9qm46wdzjr7GerLG 1BIuALUtkFMDRwZhOkQ2mNq1736NU6N22mjQJoH8Kxulw5VdLJl8J4TJBBQRwV1TeVmyd29/9s/ gOagTANSpWbrEYaueafBYdQcWwkwh4xSUewpiPaB13MHB2Eab4LvM2kMxAmosuRMYRD9QGQ4yPs 0DETxpNDGkD4mITa4F1VUkigJUYYxgKM/9TpfQ6JK0jkbewG+nkuQWOZ3irhiQUhmEKAQnwCZTh 0ERfB5MQ4JFZb/NNWU+mla3jRRzdwZHjwL6bt4R539GGZtAvk/AyMf8Pk7/pkWxnmxMKUxPZO9b 8RMiHUC0u7FbcIOBsxJSE3ruJ+SI7vYbSuZYPOeC4Rwx0WD0gAgCVnYZndUXhTG/K4K7Fzv4yir Lobo8gZIIHwZG8/v4mU2/4dYhiLc=' [7386708] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386708/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386708/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc6387749e0549ebac38038bd37cc0a0 Command failed on smithi007 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386710] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:18:18 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386710/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386710/ reached maximum tries (51) after waiting for 300 seconds [7386711] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:07:44 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386711/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386711/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e8e2f3b06be47d5bcc56a0964f33aec Command failed (workunit test fs/test_python.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7386712] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386712/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4b193d1ac8d40f0854f93e8210032e2 Host key for server 'smithi183.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPd3z/6y2UmX8a/bgkTLh9vRureYBg2xL6FqdZfDaj27', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCdhzxUo3cXBZyYSVBmhDfRqZsH7OmF+k7JA 9qjkPRxbU+Y91JtKEMvp8T6TEurnGNh+x4gocKEvod4xwobMvw8vhyuqOrGDl0ZfHiAjQDGjkys 9exp5s0y4wNCAPtnSlyGDGXDIL+Jge2IHUb4p7UIdCsHgLdPOofubTxYhgNdG/RoFGxe6j8Ubpp ZkUtaF/sbuRa7c3/oW8O39CpP1HsKUjzvQiE5KlcDcT697ev0FLUsPqGXNflktf8EHvU+3QgDUN CefW6IoOSJG2IeGTZRtPvlG/LxtUOBUg1r2Nii2v3qgpgaR8FbgPUTnEljiqgR23cGP5CZAYs3N Is0C0qXWOXcehsbqg5rgJBH/6Mx7GiTWa8WMyt5sr8wuyda7e+kxDdOsIPFLC/gKLMwrLchOa09 jIgaF1bAeFQq3Jp+jYGu8TptbmdhmbllAeK2wVIXA/dGGloqMagYMW+ToGlz6IKEpsW/XOQIwc8 ZIdoawVUOBG3taBDzd6jV/oSwBA8=' [7386713] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:30:09 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386713/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386713/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d37d646fd6b4d5c8c662c374ed9b781 saw valgrind issues [7386714] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:10:55 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386714/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386714/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d070750ae80b4316bc7b56af032eeb92 Command failed on smithi123 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386715] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386715/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386715/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a02ca67dfa6641adbbe65ce4167b8e90 Host key for server 'smithi149.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDd6duO11mRVyqhuzhxlpV+PTlHsf8mDTFEH40EmZ15+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC1PMUnG76RsAnlEz5oatc0uGC5Pw+3KrnPF p3PnRIhz6t2OYfzHltrwRgr7kawR/H0417pzi91vxReguaLYB9nsMUqSVTHICan9Sb74J//CXjv lGuy+z1v9jftPds5xsb/JBnKZR65xNynM4Pm4MAH0Z9eXmNB0N7SYkkGcbY4nbIt8SytZHcmwxU RfjGAWNnsd+DxgR66q5Yfyh3h4R5yI45VgpP2A6fcUGCD8h4jU55rysgMJxyIvt2dEfrNrOuNv4 AjAL4uf8N8SX7lcDXwysu2+KuGy6AwFwLJXruxolzavlZga2hdPsDYxzcrm6KW6X2vHDdjWpbih kDRzVVqwpUx5gNqRVv1WcXrxMixQ6MAXZNIa2Wydp/50jgv7K0cu1giFY61j4IHZZ+Srvy6HNi7 UEZzg317RSJpQ/u/ctP7xEfF02vI7JW4kl4SCSupMjMWskgG4T2pB0qs0r6/o09IpsvXnzerGkM IuBKjAokmyD/bQbEyQ/d2s6fFWfE=' [7386716] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:11:57 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386716/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386716/ "2023-09-02T01:26:22.463484+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386717] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386717/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386717/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a4e20f299e64a2cb1e36da8cd1fe064 Host key for server 'smithi184.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPd/VI4W+lOLvjJmfW0bBHjSKVEWJXhbZyxBR00rgUwL', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCfr6fuG41jS6TYteu9WpDiK7ulUs3gib2Jr JqSVwDXcp0N43qLRJprGK+sdVcwUq98Tg3aDzFTCT3+P70y0xMfT0CHc+75DuSBedhjOc5/mG3l gZPAHTztdU5nbyZbtArMkxXemE1kWMGbp5GPUPV8CHjnftBvzCDjcB2Z41Lsc4E5tITEde91wHU rEhJX6CljkT3PNYirfiZYSgnT3buY+7KvBsw5ZeX+3ROxo3nZNvuKl8o2CmDuvketi/9JGCId3J vsBJXZMUbba61XMOmBc1x85YlXQMrCJzJcVmOXhFxbUepFXvN34xTlMh0fP5EHwFSzhB/mKLRNR HMLfZHDiTW3+mN8h4qdJmEFDMNhPvF77MPl0OJ/zXO/PjttF9VtJH3VCeMu79Ra5lKLEazUBM+Q J8aWQLBtTUk4QzVQy8X7VUJ31zuWyPEFgtcCNGChym3/8Nwh0YDUrQrW6BI6Q4o2tbT99WPWniU NuwhmKUvlUrYHCz/o/PmnkB28rZM=' [7386718] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:12:50 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386718/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386718/ "2023-09-02T01:25:03.673451+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386719] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386719/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386719/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4be73f4e96c84bc5a6bdc030dc4aa47b Host key for server 'smithi077.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPMHYNG1eseoLyP3f2NP8BKIAKkSxgQ7Z/SQuh4oOwuk', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDLgF3XXo7lgtOtQn2ulSeeu+jntzJWYpqw6 mZ0DiyLIr49GqZHB8Fe8COVF4npq51pZBI/orf8yecLqZWaJHB2152VGKE0zeMT+dCpCIKcRskW JWnw+SzWDIPDOmSYLvFw+mswnC6MEbWFHdaRc31Bm3wSfdyRCJWc3oWFQjfEhPxzenSse+EULN2 m87tWyDupmkK8f8dyZLjYGsaaDFBFKT/xzeImyQ9wnosOVMgZOBxZABW5Onlfvm0AlMxQJ1cx/o /59XJf1s+By05W6uw7zf6bDEqsKVsLSRy8gQT50wlhsnsBpooLlKXjSBKIcZZQRAivak8JZVXfe ZXZbqGlIUJGd5HQyQkQvGuTdoVN1CO8mZ2dPnd6ysLPyHUgGrOjzGievwVfYmJfIJhG4pTp8W0O E/Xglv7Oj2kfgczZMwKu4C+0DvDTLw49Q/YmW9dXe+/0Cf84DaM0tAVkQjgkMMc6KRNbp+6EYRN NW9RUa0FDHsmwPDRuOHkb3CUIc88=' [7386722] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386722/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386722/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e0003f65449439a9a4620f9e7a9f25f Command failed on smithi006 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386723] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:11:17 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386723/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386723/ "2023-09-02T01:21:34.524230+0000 mon.a (mon.0) 149 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386724] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386724/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386724/ "2023-09-02T01:21:44.384789+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386725] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:17:44 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386725/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0cb9aac620049f9b6a8d7c4782588ae Command failed (workunit test fs/quota/quota.sh) on smithi194 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7386727] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386727/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386727/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19cb844b9d154cdaa1d95de5ab88defc Host key for server 'smithi144.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICscwU18doVcjsD8alpnsyedzuCDTvDGIcKH4YiZTYka', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC0QePstQJ5rzFc9uN+Z4u9iOaiTSiNh5579 frJCtuik0UickLwGb7bl8JH+PpncZI7StT25IdfU0hSiAYssJaNq3ru5apyajvphEGEcqRIRZd3 ApgN+OXPDjvTSThSSSMQjYi4zFop8hVE1ZIq4/A7Ea5sI3nW09Nu5HzpVeBA2wpyvBjvw6QxljH mxaB6kiDU0BLKb7lUHoFy0t7uvH+7/kg1efdTEnnX8rj8KTaCz2JXZfhPNnwIzbG9+DmZROkLQd TLMGZWvSkdBHb9w8eiAr0FHUN5WVcyk4F2QjaEHbJuVv8mb56qJqJ3Mh8TbFZa13lVl3heoJBKV p4g/UF41hpWmCACH62vZJUNMj7NZDvUUixIR3g4iovVFgMWC+O7odhxFw9rMpwLdPJZko6k9Gpl Hh/v0YFHk+id57mMMIYgJdy0yvxm5g7otvGQfNuDxPmFLCocZCcXxZ22MEzFI2Jboihzp/zmpp2 TKBCbLRRjaMfV3MkCHHNIUae7pmk=' [7386728] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:08:02 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386728/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386728/ "2023-09-02T01:20:42.616929+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386729] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:11:59 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386729/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386729/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1851494410294f839bfad72932ca47be Command failed on smithi008 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386731] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:24:33 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386731/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386731/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa96685d96b14b60814080a7a5f8a77a Command failed (workunit test kernel_untar_build.sh) on smithi170 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7386732] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386732/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386732/ "2023-09-02T01:17:26.072592+0000 mon.a (mon.0) 150 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386734] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:07:36 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386734/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386734/ "2023-09-02T01:15:30.824655+0000 mon.a (mon.0) 133 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386735] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386735/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386735/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9dcfdb8d4944b2abf5d081734a22392 Host key for server 'smithi110.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIBs3fimkTYa6hxggLrpNA5PsopQlseyRtaWUvl9qOE/s', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcsTNnRHZrk388jboeM/fF7p3AXlOiYYoC 21Y8X14RBVdsueYLKhN0QA7QGp4BBBP6JXB4eDO78BRwSo6ap6L4bgIPEfD4jTUdMNJ2vxPyd/R mS2NzxPnEryRwDIUS4V5EPzvfqHF+SpBYhrTv9AB2MuVpQTWNAMAkmEoWCWenaQHSoRvGQFgWjF Y4IaC6U1FMQ9eLRuAut3PMnxbJxT1iUKb9qw09+C0tlEtZEaejMikIZVLQDVmRe6p0BQ+bHpoXJ +a/BghWEXXYBVgkfN+ot61EDN3evYbIxJH/yuejf4SZC3tXUJmCAaAASiR75cWXQGBnvmijS5DQ RY6o81fxxeiF2WIqh+S1E0wJ1IH6Q26CgOuPrJKgYvrZyAR4mFK84o3Ej0iM4VLt7CeX2zXvltw 499/rUQu3WM74XbJpv0U/aHbR/F6dEG4jxihTiifwF7+wBCnbPKAYBMBbb4c5hra/2CboZDfp8h S78DMCTv3KS1VWfJgrn+DxamfzpM=' [7386736] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386736/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ad220ca989d4fc382db435951d9d05f Host key for server 'smithi152.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAID4/dYjNimwXoGxr5J0uSU3pfcTHf9kClNusASo+fzgA', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCW2xf8sc/z9FeroTpjtq1uFyfhzXb/EkPjL V3qzdHov5zPlzpHano9xd2XSSeGoBHDfeTJ6QWIS0WBohzWuILCGIuMVWN66z94tL3FlUHoAW6t 51EE8KBBJD4V/oqWISEm7Ej0i/WspNJ/kXW3nfhgDY49TcZmP7nNSre5f+aRiVU7Bbkbl75thoP qcPJrsFv3Hc4qaqeuq3I6fshGfC+ppjV0shARCfOuAISib2YSJkLtg4onohKQSmUU0zQbnTo1BF AalKEOg+s5EXCIBBO8tdWMiEPoFL4ra5P0gN43fGZ9z0IBj9jV2pN3niM9O4zDO0Wet0Ofyi9xw pKLovoaDZbDdLdMfXuVI2uZ3tOuAeEvsRvc0twTzytIEvHGcftn6JhPMcUb5Ye9RT7l5ROPpgPP 8aWAx7dsof8UTKSARCy/7lLJdJL+ls8R5Ni05CGVRzZUUdUFtIHbGKTXeUghZXXn5xHRvYMW7aG sPS59X9lAE8hTXnRofIkaX7o353E=' [7386737] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386737/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386737/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d44a35107b243a8936a13dc9ef25e45 Host key for server 'smithi046.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAID+OQ5CVMJL3iHHEAa/jxQAgt/2KdYYzYy2KkAPAQhW0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDv63Rn8WNUo8iI27vteuDXMryU4QH9ckGcP DFhTn00peQO0MbHwpF9IE+HySKsKTNUXpPfsZOaoGASfSTIUuaQKMbpVE8SbkBNSRqjp+HHrusG VInnFfdLalnDp50xPxbKWnuYuM+Rq/6UvEP+dvwEC8XoO2TZSvBp+lIsoWwoZWcpm5qZU4IidXL CY4kUf0BSly+qt2eQCBTH79UlmaOvG5ANnvA4OTCPvGQN1MZGJon7V3MFI0p4s3pSpZYes95GFW ApLETLFca4KfGfNGMQ5SKt+2eqU2O9hBpwZzYt/aRhyDZrk6HjcKTPWi4tfRgRbbR+iRWGB354Z WRBo/adW9NKVuAnjvQKS0lcb0636WzH3EhIHg6aLvRjJy62d5YzjM0r23g4YOy7vU/6UNj7oHlD gpdjl/8mrbwPJmwg0LZh7Xwsyb1j9ivTNUZ7Rozj3PF7ETSud6wEYCFxx2RoB8Pwnm5EXZwZbw4 amFKyXDGY35rWUjKX5lb2Rv9BObM=' [7386739] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:11:01 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386739/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386739/ "2023-09-02T01:21:42.800702+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386740] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386740/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f7bbaaf160a4d3a8cbc8bad40d4656b Host key for server 'smithi088.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIN1TsfUbicrUG9DyfeQ88+tz3SBWlSCS5XggOx3Qkn1f', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDEARstn4P5i4DzlVnbubfEVC06kksZLPIvY a4b02Orm6MQWuMaqtoKOk/hbIDmWyM6xeRtNb0gdi64qcnmpwV4Wj1Zw+h+AQoP7oI0jaHIg8L8 i0rpBojfkOblK52QV1umih/15rFzc/+xFn7JnxbyUiuZn2G5n4VMgpMEsmJV5K/XONYnt8nCnen SGXntdX5xfgJrlhIa+sD61MmXrm4ePteHy9MsKw9geuid+tpnJ10eI7jo6gS/wsfoWM9JTzjAxK 1h7zBF3qFDutnnznfTPaKvipY+rJRB9pLWyWB/dtka+eAzIeTMiGAJ65dWiOFyS8Al0YBFg1xnV aEy6Fl9YNZE9bfdInHbvbSQmd052Du2ri+//KFxS/s+YT+IDt7CZRHta76uc1D9+HanwchT7dQx q2P5tGVE6414xmoafEpAIj8fbX6CyNEF6GQaFOi+77yZ63wOwgmSUTuCF/bRlkjoTHYPYFsoXxq rVA4d7mJ2JNVLAQWrdYycFwHOI4s=' [7386741] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386741/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386741/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16e4b566e42e48b2919e25c5584be106 Host key for server 'smithi187.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIu+EkX9Td/A36+3DC6YwFQ/PUBVPfW8g/x0R3WiGgMg', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC47xu4iNjNpGFatkGQPGBBIjBXrw+Oh9DRf pK3rzdDWFfw3FSLwFh3LlkNqcoBpW2+g/XKd6mP69GWvLbaOZVE3tGP52ClttnfMWhScy42k/pW WXpP4gZSWhRr1RjArx3Nr3uJfUQz+3wMevtkz1sMl4MGjM+Yryx3RjtALbbCdKxM1tCpYf8qr52 8u7X2rER8OQeTZvWsPIXmbOOgh0IGMy1wvzDkyITMiouAAtBaQgwh3FaBk1hO72/Cd2vvSEDA6H 0x7smq4LBMdyl1Th7aOyqEkLQh/6nJ+TejCC/yGYEYWjco4W6kIBRzMOzvLHpZ4p36b+Qbc6ZEx ArokaMiBRc0i4JDyLIN1FCrLYekqeti/0u7ls3dXcrsbcYVLnNgzNk2GbDR0JL4axTpVwHX4qp/ 310h/Scs6Lrm0eOSCFWO0BUs8Zh1smSUe5fLGlJd0N/TieqNZBFFOqrDFDpHJm+VVINT/AZwvKh FOu0fK7A2ljXtadDCXko3tHU0+RE=' [7386742] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:01:26 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386742/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386742/ "2023-09-02T01:25:02.068372+0000 mon.a (mon.0) 115 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386743] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386743/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=308f12ebd0fc4c0a93c7f951b6917305 Command failed on smithi078 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386745] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:19:05 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386745/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386745/ "2023-09-02T01:26:26.657009+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386746] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} ----------------------------------------------------------------- time: 02:17:06 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386746/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386746/ "2023-09-02T01:27:03.771489+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386748] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:11:22 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386748/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b43fd217aa74fa892c892c13fa43bc8 Command failed on smithi059 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386749] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 07:45:27 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386749/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386749/ "2023-09-02T01:27:28.139699+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386750] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:26:55 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386750/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aac52acb697d4e08bd67153ebcb829d7 saw valgrind issues [7386751] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386751/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d40016494734857a33927fb6af3f222 Host key for server 'smithi121.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIC7lIic8ZLn28Eu79yvAjlVrY5egNkr39QTuwRxYMphH', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCsKeH+pfAgQseJeYALNyr2ZUrUZYF5bSsyh VbYpHMgBHNHtyCv22M6+pOBtfwBEpBOOfxS4GuEJGatSFGNmC07YuCuQIRD85zRzXad7m9JCJVc j6JBiWwUl5QDOOmqocR1drWTiGEilsx0F3wFnj8jupxlk3R3H+w6hmzsnvEFKRHEpJGho2px/kR xIeGFHpcs96bmQ8ymhBqVDCTwVTxUpfBawyeWJyt24ecRX1GyjwOPcDNwUDuORMkgHWTpiKAXKb MJBdDJ1uiKfhef2Mj9gA255KbOt0Eym2bX8krU+d2FiTdZto+7IqtRsrY6I3FnN2t5PdWCmJcl/ QRDmcCXGvXscj3AgdqySICzCiEzk60a4NPt7uELc2mmKNJVwZADcIsAQonovgGblD/0JshKVJBT uJ4sVLbif+Ee/usLmYFHTla+T48Ja3uvTYpIR3JsiCXVOq9q/dyd7u0m9co1sb3NN4Ynhvft2I7 lRPjIXRLEeH9DKL3EJ7S/aNS9oBs=' [7386753] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} ----------------------------------------------------------------- time: 00:12:45 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386753/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fdcf2c271a549489718a7d19655ed29 Command failed on smithi046 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386754] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:48:13 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386754/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8db8a32d92864204a12e21e104bc126d Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) [7386755] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:42:53 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386755/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386755/ "2023-09-02T01:28:07.439805+0000 mon.a (mon.0) 216 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386756] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:11:23 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386756/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aaee85dfd4ee4bba8da411dc4c969e9a Command failed on smithi019 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386757] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386757/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16bb548d8d8b4917b109bf841814f4e2 Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK1KyS3zIEFHbrpbjdQQy8eGUdMJ06f7lfyv43wkt2Fb', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCURBCz4YaXieURDkEHkLUIngSDnCFQjnniF 2D+NNeyxDKTsGQk5UUqnsJbKGe1ikRhC8sAe6b9hTZCLcx1+pVgI1ZWSoYFzBiJ0rSjLvhJEji7 kjnmdeVjvhWhiLofAmbJZFL9OcBYvt86lvkFukLYjJ1w5BCVWHNodqj3xULpZnfyUw67312CHcA lfpLQ14gRRjbFcDUTeuI/ZZMOd3rZCBB0j5oT5W9ECYNfWaeD0W1ssHQ9+XcYJt8flB/m+zRrvq 4106yr5aRrgLhM2qk6eSPJWlqbxPWFlPT8yLffTMJu7SdNBr+WvAqGi4hG1Rm6s7dT1HDBQ/hvw Yafqd7abKpfTgsbtqIffa5NdkhMDx5dGynfA8MXLxPOqXWKOyC83FAoTl097U2gJ3CcyJddtP9+ F6dakzUgMeG4bYfEfbzDHEzYTY+kOxgEvAw1vq0JQ07jgXWK+UUbysmRHbvnOV1r7qtG/Foxqfb WLPewhHFm+AE0icz9iQncRnK/aRs=' [7386759] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386759/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386759/ "2023-09-02T01:30:22.776913+0000 mon.a (mon.0) 187 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386760] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:10:36 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386760/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386760/ "2023-09-02T01:30:33.749407+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386762] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:11:02 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386762/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6a7d70ea7034626b8778eb723fea453 Command failed on smithi079 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386763] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386763/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cef05920fa947b3a1bb53c4b0c9c3e1 Host key for server 'smithi119.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIB4mkPz6WW7wmyPyep1QmctB5/QRuydQ5rfxL5DiyV6g', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCcY/ZMXh1sm488nIXOJq+3dBWeDvSeZ14C9 bSmSMyOCDAX29qFMz+2m9D97j8bGlz4brif5CJLPtLA+vBK6/yzhffZJb4oT1Y+Vo06gKURdOYD YyQA48aQ98dHM1isas73oolyguyQdD+Irtrak1HyUvIlzw9zplT74pOJwuPrsWB91ksaV4WNAJM KK8rvTzv2RjuNV2Yh0LYePV0jNLi4XeChF4pTxaKD3SqXsJKlmK6UrVT4G3vrZa4sd+0iqhJKsG EFSg858aiFmSrzhmiJkXhtgeAnGTKoVZVrIYpJXjdJkfBoHwune2MXtCd69RFIbnIYlC8IA4jzE MS10T6QQayS7jf7c42manMyVSuw3okrdDtKvm5z1/y9DkQF3tGTXL2AgJYE0r5qvNEXx/aYJDJw NPgRWN66MS5x83aMu640rmzTmvf/fItSxdyoeJn4mEn3d85XyVrppBKxQL28RlJ4zPORyKW76js mgSfTClJ5mqtO8D0p1igUY6Lu2Mk=' [7386764] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386764/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386764/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a3ee9ac58504893afb049aa06a0c21d Host key for server 'smithi155.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICIqMpbUL3JGF4AHPexGGQnBxFvGIA+PaOLyflaBSrgs', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3IVVQraQRAvogAPiZq3+iwMptfsPaUoNiL yW/73d9sqxDJgAeW0yVfU2phOCHGrG8wDIYUrs6FMmvT0cvYeFsR/hgups1bHYeE4w61Xwqs7Io x+AZEYdOzT5o06QGYhgBLiQvEbp5Iko4/aW18iQqXdwgaoX3NJDjZclcfEEduAg76vG0RqlLnHG 7osP3EMTC9hnZYgRpAVEER88cghuSGO+jDYCI0mkkuY+2iecWcVkVDllvxVwVa+WUm3FSRHxArL 2bFh1l8VsjduH12w/BG7QX1t93TdmmU0iq4PFVWlyegJ9B9M8R0ogV29HG3AszEkRsWQNvVrAFZ wJ+pOxp6av3A3pZpkGqN8oKiAuQ9Ee45o6AM9tZozTlzk6lGm8sOtV8MM1st2xMcufH0hlWZGLk 0TtmWVBQQ5d77T/lVtIcF4W3PV7bDNIaCIgf8BJfo14M00ZzcPtrf418lgLzyI0Ylp6BbxLoE8l iU/Au/uiB5321JLzxdPCivjHFXkk=' [7386767] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386767/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386767/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f792e8f8d474d639883aed15bdab576 Host key for server 'smithi121.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIN16Ld9sZ1KqNlrOpDGrk7Lkx4pe0eIMTVrgcguDmQau', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxnC8HTx2f+WtGzPoV5FFdJMelgAMhvDf8 sabyGCSJpWkteb9CKgUIpqvjEYrcaiiJ44xjCCNL51wW1XkvxKKcRDfKeK8Z6dvlMZbvU6BTfWq UDFIfj8Ufm/uWm3FIs5cbXzB5M6YQYALicHaNITAU9XnNX+ootgdb2OY8RXH6MUfq5cOAxrT9HB bjPYXW/SJUS1v8Etq6YSNjLrH8XOIiBjyL8M0pjMyb5UFfgf82m1BB0puxRLONhZB9uNMVcbHiP 64WV7uwMiIrGSwtgLxNtww54KgfJL34d1wzXe91PqlYkiY65lrkLalpK4VvfjKzACqqEVbVLy6/ 3wT/E68XqnjnU6mfiJ4XV32waD44mfzmaLwnk0JcIz7caO3ertSvkIPaNSXX8qbuMSWffexHlxE arYcN/gedhxQNcYF2nMCyAMuxvDS/pKNCNN+FliXzkXFThaJYFJRfZ65XcFFOLnokCeEkJ0E0Ni s6U1XHL4q4bRjaigrZAu5k95Tmsc=' [7386770] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:32:29 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386770/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386770/ "2023-09-02T01:51:23.043050+0000 mon.a (mon.0) 232 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386771] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:10:41 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386771/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386771/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce7d13f5a46e4de691cdf8f6cc1d9505 Command failed on smithi078 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386772] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:12:20 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386772/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386772/ "2023-09-02T01:38:33.325917+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386774] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:57 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386774/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386774/ "2023-09-02T01:39:33.508581+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386776] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:41:25 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386776/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386776/ error during scrub thrashing: rank damage found: {'backtrace'} [7386777] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:09:49 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386777/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386777/ "2023-09-02T01:47:09.281343+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386778] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386778/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a94f1068c94483ab2addf5511d55711 Host key for server 'smithi018.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHC8w76tIocNhdPexjs2ANzNo7GtzsO43y0c6avVY/dQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCg6XY8VbOOuhPvS0GtXX8P6lQhaImOFxrjL 3MFvZy7dxXXKbZGoMKF10vh2FtK1e6NUXnwGuNI6GdoZdKrDRFAShBV9tKM50RDG2v2hFM3q2Gx xM4BLD5Rb3uvdL6oxO15SIoNRNZPH5cZsKt1/kpawhZa1WEW/0A/EJtGQLRy0neUJAy2B8LQ9ns sEXJRUy6SazmjDls9bADMhCNDxZFJlZXa2KJcmVPsVv4wkq/Pzf1S9NyzI9t+k9Bgr9lLbUdHhd dTQTkA/vLIrb7Vfouyl+i85IxWx3kyEDRLGa5EwYc3q6hEZ1A7LLf29+b6KFawS2f20yCEUYqnq WxxRWYp+gmp6m0+BLh7y0q1W5smIWckRpKw3E1pqGbBdyOwcOWa7Y2FxtnyKyV2Ylx+SSPHTNJs SamsesVJsM2pLVY5+TV4P+u769C4dcq84uKbfH3RvCJJc3ET1dqniLsEZZFQ0i08VEyZkpPMJ91 7TYABlGuZSu3QAPpPLw7Z/n5/jyE=' [7386779] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:40 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386779/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9008da5ebe54501841aa80390fbc816 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7386780] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:10:26 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386780/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d2170fe4c6c4142a98d7d5bf86afbf5 Command failed on smithi081 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386781] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:41:47 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386781/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386781/ error during scrub thrashing: rank damage found: {'backtrace'} [7386782] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 00:10:56 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386782/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386782/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e026d5e292d4e34a6bc773c55763e87 Command failed on smithi008 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" [7386783] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386783/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f1598d2ea8843b293fb63d7e2e98620 Host key for server 'smithi149.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINIdCgHQ2nH3lZl9ZbfCuhSotDyNNW7ra0i6G1jb0I25', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCZc7ISI9fnHl0DohWhcEz3dYUZxAvKV/hUj 5k/YhasC02kWJEuLqcks1zFyHxmI3i1Ij/ahpevREmdY6pXlsFe7AcBr8ctlZL39TeF4t9ETv6p lCUN4VL+amqLHazDV8t4HNw1e/cUoctoNHtwWZ4P2PUvx545DACpcvMzq36qQwQkLEIX2wUrnge XelA3GfzTJrmzYD7YSzXHFFajaWbyYMMCrie2yr7gSTIFLpGp5nTdAqZyRJSAkql0FgXiTULKLF 9apWOuIGSddDlCPje+UEfANBtwgXmZJI8hg1vQMn9ivzfjQPWG+Z/lVIjtLz73LEgO/jSQJW6wp xfalNJJWra0SgnHISJZozN/U0ZI5zZk6eQMPS96zEafGtU5gf49h020fXxPtDfgtBva6ouLaKDn 7JxQfjEChx3i514UrzELLUCfiZxnC0De7LKHK9+oLny3/fZDNiaiT00kPY7SflEeO4YTumltIb4 wD0vHt5t8CWsZ0rbDGO/zbf+7dmM=' [7386784] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:26:10 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386784/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386784/ "2023-09-02T01:48:52.748633+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386786] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{test}} ----------------------------------------------------------------- time: 00:32:10 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386786/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386786/ "2023-09-02T01:58:14.930322+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386787] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386787/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0023127069c04a57bf75773da39e911c Host key for server 'smithi114.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK5p6LrQe30wfff0x8diOanKdjYqjcVGEW2LQyLTohNK', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDiTc4rcTiS+e9Ie0cWOQm0W8ap+L+GTGj4h WnfIOY9SvIJbIzKeoFSPFLxIpJ1iWxRuaPa+wzUi/7M5ClwC/Ehe76xzQhEARRcK+84Ws5+CDdz DHL/OynFrAonkiTqIFEsGZB0Je0CIbmc8BRSADRBYBWMuzVceRbQl4g8yjdj9NN9uX5M5GmX3Bu zqCki5mVe7YbkntN8iSQHY25lnZ/QwQpLWb1G9XUMv0dt38upoATRe/zStvMdrUCuySFO5XhkG3 H2upclIbPDM2FYIBs3x8aUil9VYzi5Jyl9Y4uBAjKFgCdJSTuihgtkSEUops/7xTZ1+V55OZ9oI f0C1d1V99vZ1q88/0FMRIBPuAnOQbx6BowBbaquPJGz6VkYRDfCqTHbeQA+OtfFDQM2R+5QCm3k lKw/sOt974CpjYDfN9o8ivUQn4p9efqWUCVhFTwyY71kK/vzEHx7wr9M6re414NqvRW22AiaZ7h iXRHfNLPkLW/RQ2Zoe+vx7Xxclfk=' [7386788] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:06:22 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386788/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386788/ "2023-09-02T01:50:58.183742+0000 mon.a (mon.0) 220 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386789] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:19:52 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386789/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386789/ reached maximum tries (51) after waiting for 300 seconds [7386790] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:08:40 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386790/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386790/ "2023-09-02T01:50:33.098467+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7386791] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386791/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6e548b3dfca447395ff72e5642b510e Command failed on smithi038 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600" Dead ================================================================= [7386704] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386704/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386704/ hit max job timeout [7386706] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386706/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386706/ hit max job timeout [7386721] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386721/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386721/ hit max job timeout [7386738] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386738/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386738/ hit max job timeout [7386768] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386768/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386768/ hit max job timeout [7386769] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386769/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386769/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7386692] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:40:38 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386692/ [7386694] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:34:20 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386694/ [7386697] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:14:30 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386697/ [7386701] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:40:17 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386701/ [7386707] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:10:55 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386707/ [7386709] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:28:37 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386709/ [7386720] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:51:07 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386720/ [7386726] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:45:04 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386726/ [7386730] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:51:17 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386730/ [7386733] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:12 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386733/ [7386744] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:38:03 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386744/ [7386747] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:09:21 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386747/ [7386752] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:15:16 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386752/ [7386758] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:14:08 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386758/ [7386761] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 01:30:46 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386761/ [7386765] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:37:40 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386765/ [7386766] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:17:26 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386766/ [7386773] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:46:22 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386773/ [7386775] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:18 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386775/ [7386785] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:44 info: http://pulpito.ceph.com/pdonnell-2023-09-02_01:04:01-fs-wip-batrick-testing-20230830.165602-distro-default-smithi/7386785/