2022-04-10T00:41:46.526 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-10T00:41:46.528 INFO:teuthology.results:seed: '2426' 2022-04-10T00:50:40.966 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-10T00:50:40.970 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-10T01:18:42.810 INFO:teuthology.results:Tests finished! gathering results... 2022-04-10T01:18:43.259 INFO:scrape:Found 7 jobs 2022-04-10T01:18:43.692 INFO:scrape:Found 1 distinct failure reasons 2022-04-10T01:18:43.693 INFO:scrape:Failure: client still has session: {'id': 4463, 'entity': {'name': {'type': 'client', 'num': 4463}, 'addr': {'type': 'v1', 'addr': '172.21.15.169:0', 'nonce': 1804096549}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 24.04191595, 'requests_in_flight': 0, 'num_completed_requests': 0, 'num_completed_flushes': 1, 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.9706233577985948, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.4463 v1:172.21.15.169:0/1804096549', 'completed_requests': [], 'prealloc_inos': [{'start': '0x10000000004', 'length': 500}, {'start': '0x10000007789', 'length': 974}], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x00000000000003ff'}}, 'ceph_sha1': 'ac978832a85a67de60a49b34ed497d7075f4b151', 'ceph_version': 'ceph version 16.2.7-864-gac978832 (ac978832a85a67de60a49b34ed497d7075f4b151) pacific (stable)', 'entity_id': '0', 'hostname': 'smithi169', 'mount_point': '/home/ubuntu/cephtest/mnt.0', 'pid': '28994', 'root': '/'}} 2022-04-10T01:18:43.693 INFO:scrape:4 jobs: ['6784687', '6784686', '6784683', '6784684'] 2022-04-10T01:18:43.694 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'pg-warn', 'tasks/{0-pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-10T01:18:43.694 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-10T01:18:43.695 INFO:scrape: 2022-04-10T01:18:43.696 INFO:teuthology.results:Sending results to rraja@redhat.com: Test Run: rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [6784683] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:14:08 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784683/ log: http://qa-proxy.ceph.com/teuthology/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784683/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6b01e391ab441eab2cb7745507429bf client still has session: {'id': 4638, 'entity': {'name': {'type': 'client', 'num': 4638}, 'addr': {'type': 'v1', 'addr': '172.21.15.195:0', 'nonce': 1360750671}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 107.049314476, 'requests_in_flight': 0, 'num_completed_requests': 0, 'num_completed_flushes': 0, 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.7826823948565849, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.4638 v1:172.21.15.195:0/1360750671', 'completed_requests': [], 'prealloc_inos': [], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x00000000000003ff'}}, 'ceph_sha1': 'ac978832a85a67de60a49b34ed497d7075f4b151', 'ceph_version': 'ceph version 16.2.7-864-gac978832 (ac978832a85a67de60a49b34ed497d7075f4b151) pacific (stable)', 'entity_id': '1', 'hostname': 'smithi195', 'mount_point': '/home/ubuntu/cephtest/mnt.1', 'pid': '28976', 'root': '/'}} [6784684] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:18:55 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784684/ log: http://qa-proxy.ceph.com/teuthology/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784684/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7ca94603e374f6987e50cc0471263cf client still has session: {'id': 4490, 'entity': {'name': {'type': 'client', 'num': 4490}, 'addr': {'type': 'v1', 'addr': '172.21.15.119:0', 'nonce': 3298786548}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 19.971495406, 'requests_in_flight': 0, 'num_completed_requests': 0, 'num_completed_flushes': 542, 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.9704625364656982, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.4490 v1:172.21.15.119:0/3298786548', 'completed_requests': [], 'prealloc_inos': [{'start': '0x10000000004', 'length': 500}, {'start': '0x10000001bce', 'length': 402}, {'start': '0x10000001f55', 'length': 501}], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x00000000000003ff'}}, 'ceph_sha1': 'ac978832a85a67de60a49b34ed497d7075f4b151', 'ceph_version': 'ceph version 16.2.7-864-gac978832 (ac978832a85a67de60a49b34ed497d7075f4b151) pacific (stable)', 'entity_id': '0', 'hostname': 'smithi119', 'mount_point': '/home/ubuntu/cephtest/mnt.0', 'pid': '29030', 'root': '/'}} [6784686] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:13:33 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784686/ log: http://qa-proxy.ceph.com/teuthology/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784686/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e6ede5d72e1c46c7bf5e90b555d4fd08 client still has session: {'id': 4471, 'entity': {'name': {'type': 'client', 'num': 4471}, 'addr': {'type': 'v1', 'addr': '172.21.15.123:0', 'nonce': 4093450222}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 98.996651865, 'requests_in_flight': 0, 'num_completed_requests': 0, 'num_completed_flushes': 0, 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.7973772861617758, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.4471 v1:172.21.15.123:0/4093450222', 'completed_requests': [], 'prealloc_inos': [], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x00000000000003ff'}}, 'ceph_sha1': 'ac978832a85a67de60a49b34ed497d7075f4b151', 'ceph_version': 'ceph version 16.2.7-864-gac978832 (ac978832a85a67de60a49b34ed497d7075f4b151) pacific (stable)', 'entity_id': '1', 'hostname': 'smithi123', 'mount_point': '/home/ubuntu/cephtest/mnt.1', 'pid': '29025', 'root': '/'}} [6784687] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:18:20 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784687/ log: http://qa-proxy.ceph.com/teuthology/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784687/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e10715f9852456b940b2608500331c5 client still has session: {'id': 4463, 'entity': {'name': {'type': 'client', 'num': 4463}, 'addr': {'type': 'v1', 'addr': '172.21.15.169:0', 'nonce': 1804096549}}, 'state': 'open', 'num_leases': 0, 'num_caps': 1, 'request_load_avg': 0, 'uptime': 24.04191595, 'requests_in_flight': 0, 'num_completed_requests': 0, 'num_completed_flushes': 1, 'reconnecting': False, 'recall_caps': {'value': 0, 'halflife': 60}, 'release_caps': {'value': 0, 'halflife': 60}, 'recall_caps_throttle': {'value': 0, 'halflife': 1.5}, 'recall_caps_throttle2o': {'value': 0, 'halflife': 0.5}, 'session_cache_liveness': {'value': 0.9706233577985948, 'halflife': 300}, 'cap_acquisition': {'value': 0, 'halflife': 10}, 'delegated_inos': [], 'inst': 'client.4463 v1:172.21.15.169:0/1804096549', 'completed_requests': [], 'prealloc_inos': [{'start': '0x10000000004', 'length': 500}, {'start': '0x10000007789', 'length': 974}], 'client_metadata': {'client_features': {'feature_bits': '0x000000000000ffff'}, 'metric_spec': {'metric_flags': {'feature_bits': '0x00000000000003ff'}}, 'ceph_sha1': 'ac978832a85a67de60a49b34ed497d7075f4b151', 'ceph_version': 'ceph version 16.2.7-864-gac978832 (ac978832a85a67de60a49b34ed497d7075f4b151) pacific (stable)', 'entity_id': '0', 'hostname': 'smithi169', 'mount_point': '/home/ubuntu/cephtest/mnt.0', 'pid': '28994', 'root': '/'}} Pass ================================================================= [6784681] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-compat_client/no}} time: 00:25:03 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784681/ [6784682] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-upgrade}} time: 00:09:10 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784682/ [6784685] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-pacific 1-client 2-upgrade 3-compat_client/no}} time: 00:25:35 info: http://pulpito.ceph.com/rraja-2022-04-10_00:40:48-fs-rraja-wip-53996-distro-default-smithi/6784685/