2023-05-13T01:42:11.125 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-13T01:42:11.126 INFO:teuthology.results:subset: '111/12000' 2023-05-13T01:42:11.126 INFO:teuthology.results:seed: '7940' 2023-05-13T02:10:57.758 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-13T02:10:57.778 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-13T03:31:14.330 INFO:teuthology.results:Tests finished! gathering results... 2023-05-13T03:31:22.102 INFO:scrape:Found 15 jobs 2023-05-13T03:31:22.103 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272964/teuthology.log 2023-05-13T03:31:22.728 INFO:scrape:Found 5 distinct failure reasons 2023-05-13T03:31:22.729 INFO:scrape:Dead: None 2023-05-13T03:31:22.729 INFO:scrape:2 jobs: ['7272964', '7272966'] 2023-05-13T03:31:22.729 INFO:scrape: 2023-05-13T03:31:22.730 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1683946086.087:44): avc: denied { node_bind } for pid=1626 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2023-05-13T03:31:22.730 INFO:scrape:4 jobs: ['7272970', '7272958', '7272952', '7272976'] 2023-05-13T03:31:22.730 INFO:scrape:suites intersection: ['ignorelist_health', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/rhel_8', 'thrashosds-health}'] 2023-05-13T03:31:22.731 INFO:scrape:suites union: ['ignorelist_health', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/rhel_8', 'tasks/admin_socket_objecter_requests', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/rados_api_tests', 'tasks/snaps-few-objects', 'thrashosds-health}'] 2023-05-13T03:31:22.731 INFO:scrape: 2023-05-13T03:31:22.731 INFO:scrape:Failure: SSH connection to smithi121 was lost: 'sudo yum install -y kernel' 2023-05-13T03:31:22.732 INFO:scrape:1 jobs: ['7272974'] 2023-05-13T03:31:22.732 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-comp-zstd', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_8', 'tasks/readwrite', 'thrashosds-health}'] 2023-05-13T03:31:22.733 INFO:scrape: 2023-05-13T03:31:22.733 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds 2023-05-13T03:31:22.733 INFO:scrape:1 jobs: ['7272972'] 2023-05-13T03:31:22.733 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-comp-zlib', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/radosbench', 'thrashosds-health}'] 2023-05-13T03:31:22.734 INFO:scrape: 2023-05-13T03:31:22.734 INFO:scrape:Failure: failed to install new kernel version within timeout 2023-05-13T03:31:22.734 INFO:scrape:1 jobs: ['7272968'] 2023-05-13T03:31:22.735 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-comp-lz4', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/centos_8', 'tasks/cfuse_workunit_suites_truncate_delay', 'thrashosds-health}'] 2023-05-13T03:31:22.735 INFO:scrape: 2023-05-13T03:31:22.736 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/ failed: 6 dead: 3 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7272952] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/rhel_8 tasks/admin_socket_objecter_requests thrashosds-health} ----------------------------------------------------------------- time: 00:24:12 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272952/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272952/ SELinux denials found on ubuntu@smithi070.front.sepia.ceph.com: ['type=AVC msg=audit(1683943853.053:44): avc: denied { node_bind } for pid=3835 comm="ping" saddr=172.21.15.70 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7272958] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zstd powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_ffsb thrashosds-health} ----------------------------------------------------------------- time: 00:40:38 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272958/ SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1683944348.971:45): avc: denied { node_bind } for pid=3154 comm="ping" saddr=172.21.15.102 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7272968] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_truncate_delay thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272968/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272968/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a41d2250e27e4bf89120a40c2735be9d failed to install new kernel version within timeout [7272970] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/rhel_8 tasks/rados_api_tests thrashosds-health} ----------------------------------------------------------------- time: 00:41:47 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272970/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272970/ SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1683946086.087:44): avc: denied { node_bind } for pid=1626 comm="ping" saddr=172.21.15.27 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7272972] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/ubuntu_latest tasks/radosbench thrashosds-health} ----------------------------------------------------------------- time: 01:10:36 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272972/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4690690478754d309d30810e653c014c reached maximum tries (500) after waiting for 3000 seconds [7272976] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-all-distro/rhel_8 tasks/snaps-few-objects thrashosds-health} ----------------------------------------------------------------- time: 00:44:56 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272976/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272976/ SELinux denials found on ubuntu@smithi131.front.sepia.ceph.com: ['type=AVC msg=audit(1683946567.748:45): avc: denied { node_bind } for pid=3125 comm="ping" saddr=172.21.15.131 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] Dead ================================================================= [7272964] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-stupid powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_fsync thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272964/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272964/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7272966] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-bitmap powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_pjd thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272966/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272966/ Error reimaging machines: Failed to power on smithi086 [7272974] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zstd powercycle/default supported-all-distro/centos_8 tasks/readwrite thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272974/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8dca607faf8f47d8ac79930c9697d9a3 SSH connection to smithi121 was lost: 'sudo yum install -y kernel' Pass ================================================================= [7272950] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-bitmap powercycle/default supported-all-distro/centos_8 tasks/snaps-many-objects thrashosds-health} time: 00:32:38 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272950/ [7272954] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_kernel_untar_build thrashosds-health} time: 00:47:12 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272954/ [7272956] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_misc thrashosds-health} time: 00:36:55 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272956/ [7272960] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_fsstress thrashosds-health} time: 00:22:36 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272960/ [7272962] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-low-osd-mem-target powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_fsx thrashosds-health} time: 00:33:18 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272962/ [7272978] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-low-osd-mem-target powercycle/default supported-all-distro/ubuntu_latest tasks/snaps-many-objects thrashosds-health} time: 00:32:03 info: http://pulpito.ceph.com/yuriw-2023-05-12_19:29:18-powercycle-reef-distro-default-smithi/7272978/