2022-12-03T01:22:35.677 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-03T01:22:35.678 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_lflores@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/master} --archive /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687 --verbose --name lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi -- /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/orig.config.yaml 2022-12-03T01:22:35.707 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-12-03T01:22:35.776 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687 branch: wip-yuri7-testing-2022-11-18-1548 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/master} email: null first_in_suite: false job_id: '7101687' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri7-testing-2022-11-18-1548 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 kernel: hwe: true kubeadm: pod_network: calico version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri7-testing-2022-11-18-1548 sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri7-testing-2022-11-18-1548 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 targets: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2WxWvWjADzCCGdPLzJAfBVCPIIeRZMhVKgcnENlSq4dXWPd5XnvFsx6dFcOgiuL4JN6T93GwZBuzhTI77xNlNNamDJ4IHDi6OmNlg2z56M3SmeYXNY21V/Egshvvg7FS+ZZ3MQw5g3DIM5ryyE+is5kh4nbzBtuALmr9nMp4/oTcIOUcf/EJk67r9IPdDhhv3EAqmiI+37Iqk803prTb3mpb54iUutAXtFMsm2+6OgiUlmEmJxukKCtkx9adv/1iv6eokimTY4khDueSr+0GuR5nWLdG3mPRbASSERoyQJ83r2ttaeHJB8fB7B6lFvK1c/a6kN5YBH6ZYbPr56GDAuAlbhw7Rr7ze1RLEdNtYW7k8mhWsR33EKYEhhLFiZau/dWqEbaWb1xRHL6czWZd90UOJtTxyDNdgVh61/vvl0kIib3IYPX9ZmEZuou+hZ2apgoAit94cNQ2rGu4kXCuB/OUvbJXmfpospaGYoj5J98X0djV+yQ9CKULdbCHOkCE= smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTYaYroROwpyrYbKS3rK2RLDVtuSNcELmQ+hu7NtGWHZlZDEEWJZEKK4MS6IFBrKUXxXMOto22yjBfBQ6AWDuoXXVGSNN6/cTCfwVGya0MuedcKvM2XNF42DrJjO1M9YX6RbHYLY4lRF5UHWUGF87L7ilrrdvJkxAPzbc+1LsLLALpb8GJQ9tOXbEuosBGImmvxQQ4yfkxkrEEjNRctnD3o9M9e/is9ZxN0WT2+OzzTmMMRAs0Fc8SJ4XfHHVFFYsOhHBjwleqiQj1m0DdnTed/yQd502sDuwvP7Mv08ACRfbbl3q2Dw+tI4XHVu7Iz08ThE93iAwoJnnuFtRxLTB2ccI0rwhlnOOQgEKQBJUV5CcXolUoukwEr2apei2gQBlMGvxwO8qjbUtFLHP/IT70GCAv5tJxIz7fOtTcriV+xU2LHMhfJsQ76l680PwNa9l7P0k54jHs8vAVxYvTZt+oat6smaVg5VHJPRT605Nr6h3zh5YjIp0s8lkYAWp8uwk= smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjpaEPYI6dhOtNNDoVaKsI709Lv2pjzuu/WTfJBz1r9VXRwpX9+6KOvrQrWGBHysmvw5PFy8F6garVKz6DKm8xHsDYdfbIMPxR2jzKsrBY8pCMKl7/3yIJQCLP5oX/Cn4pltJyJhIsvdHRPiaCEybLuXsIs8BUcKqI82rDehvZrAb2yco4WdoAPtW+7B3CahnjQu76zX6Q1z35Vq/StCm02sGnIipqF7X6vwnY/U1DsgqH1sdCGynlhRTsz1sgbBv8m804jZMJKGWAEnFqR+OGkPLBcvP0/E4hHk6Ha3AB82odBR+kUHShR0pikcEjIfO5gRC1ZRZY3gp4KZg/C+eb4tvMx9I68BInRpSft3R4coh2ElNkFUIY6kJrAQMTllAt4cZqKDqGtrzDqxaam00K+QAqAtg+HeKaHK4Tv/QwZWgX+JLlJpQak1zuAEXMWBf1efMse3r082zm03Dxtnv5JNI80eXAkH/ocK5DJv3JJ64XVO3evo9IbI+MlUoFR2E= tasks: - kubeadm: null - nvme_loop: null - rook: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 timestamp: 2022-12-02_15:43:15 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23126 2022-12-03T01:22:35.777 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa; will attempt to use it 2022-12-03T01:22:35.777 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks 2022-12-03T01:22:35.778 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-12-03T01:22:35.780 INFO:teuthology.task.internal:Checking packages... 2022-12-03T01:22:35.805 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '48d203d9b836cc224eb4085f6198d948c6fa1bb1' 2022-12-03T01:22:35.805 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-12-03T01:22:35.806 INFO:teuthology.packaging:ref: None 2022-12-03T01:22:35.806 INFO:teuthology.packaging:tag: None 2022-12-03T01:22:35.807 INFO:teuthology.packaging:branch: wip-yuri7-testing-2022-11-18-1548 2022-12-03T01:22:35.807 INFO:teuthology.packaging:sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 2022-12-03T01:22:35.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri7-testing-2022-11-18-1548 2022-12-03T01:22:35.966 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-872-g48d203d9-1focal 2022-12-03T01:22:35.967 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-12-03T01:22:35.985 INFO:teuthology.task.internal:no buildpackages task found 2022-12-03T01:22:35.986 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-12-03T01:22:36.018 INFO:teuthology.task.internal:Saving configuration 2022-12-03T01:22:36.044 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-12-03T01:22:36.076 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:22:36.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.111414', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINq1u5hYDvgWaM2uDbBwTMulbPPN13NpdULE6alu3Yni'} 2022-12-03T01:22:36.132 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.109816', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBRA2JzHzqwXSgWTQ5/zle/URh+4gliQUodqLpxPF5GE'} 2022-12-03T01:22:36.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.112514', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJTdESnxaM3B9VA3H8igDFkST8wEUZmhkT7KZN4IMnaQ'} 2022-12-03T01:22:36.166 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-12-03T01:22:36.173 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-12-03T01:22:36.174 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-12-03T01:22:36.174 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-12-03T01:22:36.175 INFO:teuthology.run_tasks:Running task console_log... 2022-12-03T01:22:36.354 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f476887c2f0>, signals=[15]) 2022-12-03T01:22:36.356 INFO:teuthology.run_tasks:Running task internal.connect... 2022-12-03T01:22:36.365 INFO:teuthology.task.internal:Opening connections... 2022-12-03T01:22:36.365 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2022-12-03T01:22:36.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:22:36.486 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2022-12-03T01:22:36.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:22:36.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2022-12-03T01:22:36.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:22:36.741 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-12-03T01:22:36.749 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2022-12-03T01:22:36.758 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2022-12-03T01:22:36.759 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2022-12-03T01:22:36.810 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2022-12-03T01:22:36.811 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:22:36.812 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2022-12-03T01:22:36.812 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2022-12-03T01:22:36.812 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:22:36.812 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="20.04" 2022-12-03T01:22:36.813 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:22:36.813 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:22:36.814 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:22:36.814 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:22:36.814 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=focal 2022-12-03T01:22:36.815 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:22:36.816 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2022-12-03T01:22:36.844 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2022-12-03T01:22:36.856 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2022-12-03T01:22:36.857 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2022-12-03T01:22:36.903 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2022-12-03T01:22:36.904 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:22:36.904 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2022-12-03T01:22:36.904 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2022-12-03T01:22:36.904 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:22:36.905 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2022-12-03T01:22:36.905 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:22:36.905 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:22:36.905 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:22:36.906 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:22:36.907 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2022-12-03T01:22:36.907 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:22:36.908 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2022-12-03T01:22:36.947 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2022-12-03T01:22:36.956 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2022-12-03T01:22:36.957 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2022-12-03T01:22:37.005 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Ubuntu" 2022-12-03T01:22:37.006 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:22:37.006 INFO:teuthology.orchestra.run.smithi155.stdout:ID=ubuntu 2022-12-03T01:22:37.006 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE=debian 2022-12-03T01:22:37.006 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:22:37.007 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="20.04" 2022-12-03T01:22:37.007 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:22:37.007 INFO:teuthology.orchestra.run.smithi155.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:22:37.007 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:22:37.008 INFO:teuthology.orchestra.run.smithi155.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:22:37.008 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_CODENAME=focal 2022-12-03T01:22:37.008 INFO:teuthology.orchestra.run.smithi155.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:22:37.009 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2022-12-03T01:22:37.080 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-12-03T01:22:37.091 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-12-03T01:22:37.135 INFO:teuthology.task.internal:Checking for old test directory... 2022-12-03T01:22:37.136 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:22:37.138 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:22:37.142 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:22:37.149 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-12-03T01:22:37.159 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-12-03T01:22:37.160 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:22:37.163 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:22:37.167 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:22:37.199 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-12-03T01:22:37.298 INFO:teuthology.run_tasks:Running task kernel... 2022-12-03T01:22:37.314 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-12-03T01:22:37.315 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-12-03T01:22:37.315 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-12-03T01:22:37.315 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-12-03T01:22:37.316 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:22:37.317 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:22:37.317 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:22:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:22:37.324 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2022-12-03T01:22:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:22:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:22:37.327 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2022-12-03T01:22:37.328 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2022-12-03T01:22:37.337 INFO:teuthology.orchestra.run.smithi137.stdout:5.4.0-128-generic 2022-12-03T01:22:37.338 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-128-generic 2022-12-03T01:22:37.338 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.4.0-128-generic 2022-12-03T01:22:37.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2022-12-03T01:22:37.340 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.4.0-128-generic 2022-12-03T01:22:37.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2022-12-03T01:22:37.388 INFO:teuthology.orchestra.run.smithi155.stdout:5.4.0-128-generic 2022-12-03T01:22:37.388 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.4.0-128-generic 2022-12-03T01:22:37.389 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2022-12-03T01:22:37.492 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2022-12-03T01:22:37.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2022-12-03T01:22:37.513 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2022-12-03T01:22:37.657 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:22:37.683 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:22:37.684 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:22:37.684 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:22:37.685 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:22:37.722 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:22:37.768 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:22:37.787 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:22:37.848 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:22:37.866 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:22:37.959 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:22:38.016 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:22:38.039 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:22:38.040 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:22:38.112 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:22:38.114 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:22:38.115 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:22:38.117 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:22:38.130 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:22:38.141 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:22:38.152 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:22:38.161 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:22:38.162 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:22:38.167 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:22:38.168 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:22:38.171 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:22:38.173 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:22:38.173 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:22:38.174 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:22:38.182 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:22:38.184 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:22:38.188 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:22:38.188 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:22:38.189 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:22:38.193 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:22:38.194 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:22:38.197 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:22:38.198 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:22:38.206 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:22:38.207 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:22:38.208 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:22:38.208 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:22:38.213 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:22:38.214 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:22:38.214 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:22:38.215 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:22:38.219 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:22:38.247 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:22:38.268 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:22:38.269 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:22:38.270 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:22:38.270 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:22:38.272 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:22:38.277 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:22:38.278 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:22:38.279 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:22:38.297 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:22:38.300 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.301 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:22:38.302 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:22:38.304 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:22:38.305 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:22:38.306 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:22:38.314 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:22:38.315 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:22:38.315 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:22:38.316 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:22:38.316 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:22:38.321 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:22:38.325 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.325 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:22:38.326 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:22:38.326 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:22:38.336 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:22:38.341 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:22:38.343 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:22:38.343 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:22:38.344 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:22:38.345 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:22:38.345 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:22:38.347 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:22:38.347 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:22:38.348 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:22:38.356 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:22:38.359 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:22:38.682 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:22:38.700 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:22:38.712 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:22:38.726 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:22:38.726 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:22:38.732 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:22:38.744 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:22:38.745 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:22:38.746 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:22:38.766 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:22:38.766 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:22:38.771 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:22:38.785 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:22:38.790 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.791 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:22:38.797 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:22:38.815 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:22:38.816 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:22:38.835 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:22:38.841 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:22:38.842 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:22:38.843 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:22:38.865 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:22:38.865 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:22:38.865 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:22:38.866 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:22:38.896 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:22:38.898 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.899 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:22:38.900 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:22:38.923 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:22:38.924 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:22:38.924 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:22:38.935 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:22:38.935 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:22:38.936 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:22:38.936 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:22:38.937 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:22:38.943 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:22:39.000 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:22:39.001 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:22:39.019 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:22:39.020 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:22:42.913 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 13.5 MB in 2s (6,397 kB/s) 2022-12-03T01:22:42.935 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 13.5 MB in 2s (6,819 kB/s) 2022-12-03T01:22:42.942 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 13.5 MB in 2s (6,599 kB/s) 2022-12-03T01:22:44.231 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:22:44.253 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:22:44.266 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:22:44.290 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:22:44.331 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:22:44.335 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:22:44.358 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:22:44.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:22:44.430 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:22:44.516 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:22:44.517 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:22:44.541 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:22:44.542 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:22:44.614 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:22:44.614 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:22:44.681 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2022-12-03T01:22:44.681 INFO:teuthology.orchestra.run.smithi155.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.682 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:22:44.682 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:22:44.684 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2022-12-03T01:22:44.684 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:22:44.684 INFO:teuthology.orchestra.run.smithi155.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:22:44.695 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2022-12-03T01:22:44.696 INFO:teuthology.orchestra.run.smithi142.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.696 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:22:44.696 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:22:44.697 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2022-12-03T01:22:44.697 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:22:44.697 INFO:teuthology.orchestra.run.smithi142.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:22:44.733 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-12-03T01:22:44.733 INFO:teuthology.orchestra.run.smithi155.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.734 INFO:teuthology.orchestra.run.smithi155.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:22:44.735 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:22:44.735 INFO:teuthology.orchestra.run.smithi155.stdout: thermald upower usbmuxd 2022-12-03T01:22:44.748 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-12-03T01:22:44.748 INFO:teuthology.orchestra.run.smithi142.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.748 INFO:teuthology.orchestra.run.smithi142.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:22:44.749 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:22:44.750 INFO:teuthology.orchestra.run.smithi142.stdout: thermald upower usbmuxd 2022-12-03T01:22:44.776 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2022-12-03T01:22:44.777 INFO:teuthology.orchestra.run.smithi137.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.777 INFO:teuthology.orchestra.run.smithi137.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:22:44.778 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:22:44.778 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2022-12-03T01:22:44.779 INFO:teuthology.orchestra.run.smithi137.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:22:44.780 INFO:teuthology.orchestra.run.smithi137.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:22:44.787 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:22:44.787 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:22:44.788 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:22:44.788 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:22:44.806 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:22:44.807 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:22:44.808 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:22:44.808 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:22:44.840 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-12-03T01:22:44.840 INFO:teuthology.orchestra.run.smithi137.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:22:44.841 INFO:teuthology.orchestra.run.smithi137.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:22:44.841 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:22:44.842 INFO:teuthology.orchestra.run.smithi137.stdout: thermald upower usbmuxd 2022-12-03T01:22:44.844 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:22:44.850 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:22:44.853 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:22:44.869 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:22:44.875 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:22:44.878 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:22:44.882 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:22:44.885 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:22:44.903 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:22:44.911 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:22:45.021 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:22:45.021 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:22:45.021 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:22:45.021 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:22:45.161 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:22:45.266 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:22:45.268 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:22:45.299 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:22:45.326 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:22:45.343 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:22:45.384 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:22:45.414 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:22:45.448 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:22:45.973 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:22:45.973 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:22:46.046 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:22:46.057 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:22:46.058 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:22:46.246 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:22:46.247 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:22:46.251 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:22:46.252 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:22:46.255 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:22:46.337 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:22:46.449 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 90.3 MB in 1s (68.7 MB/s) 2022-12-03T01:22:46.623 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:22:46.647 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 90.3 MB in 1s (60.3 MB/s) 2022-12-03T01:22:46.657 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:22:46.832 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:22:47.955 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:22:47.960 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:22:48.081 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:22:48.290 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:22:48.294 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:22:48.440 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:22:48.457 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:22:48.489 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:22:48.490 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:22:48.495 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:22:48.496 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:22:48.498 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:22:48.507 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:22:48.561 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:22:48.836 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:22:48.849 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:22:48.864 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:22:48.877 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:22:48.893 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:22:48.895 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 90.3 MB in 4s (24.7 MB/s) 2022-12-03T01:22:48.922 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:22:49.107 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:22:49.163 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:22:49.177 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:22:49.213 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:22:49.233 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:22:49.248 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:22:49.293 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:22:49.558 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:22:49.572 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:22:49.610 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:22:49.636 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:22:49.649 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:22:49.698 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:22:49.936 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:22:49.952 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:49.986 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:49.988 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:22:50.002 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:22:50.031 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:22:50.350 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:22:50.366 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:50.401 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:22:50.406 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:22:50.408 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:50.553 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:22:50.924 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:22:50.935 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:22:50.972 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:22:51.309 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:22:51.322 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:22:51.359 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:22:51.670 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:22:51.683 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:22:51.720 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:22:52.082 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:22:52.095 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:22:52.141 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:22:52.331 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:22:52.347 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:52.449 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:52.500 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:22:52.515 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:52.559 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:52.799 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:22:52.817 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:52.902 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:52.917 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:22:52.930 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:52.967 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:53.452 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:22:53.465 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:53.510 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:54.976 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:22:54.989 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:55.076 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:55.555 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:22:55.575 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:22:55.596 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:22:59.625 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:22:59.638 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:22:59.672 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:22:59.815 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:22:59.835 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:22:59.873 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:22:59.991 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:23:00.007 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:23:00.050 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:00.151 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:23:00.172 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:23:00.200 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:00.420 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package thermald. 2022-12-03T01:23:00.434 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:23:00.495 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:00.561 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package thermald. 2022-12-03T01:23:00.577 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:23:00.620 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:00.983 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package upower. 2022-12-03T01:23:00.997 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:23:01.051 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:23:01.166 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package upower. 2022-12-03T01:23:01.180 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:23:01.234 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:23:01.471 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:23:01.485 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:23:01.529 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:01.595 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:23:01.610 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:23:01.645 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:01.814 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:23:01.934 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:23:01.949 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:23:02.060 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:23:02.074 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:23:02.122 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:23:02.140 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:23:02.189 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:23:02.189 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:23:02.195 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:23:02.304 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:23:02.318 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:23:02.430 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:02.444 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:23:02.482 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:23:02.496 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:23:02.540 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:02.564 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:23:02.569 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:02.721 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:23:02.893 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package thermald. 2022-12-03T01:23:02.907 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:23:02.951 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:03.000 INFO:teuthology.orchestra.run.smithi155.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:03.049 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:03.214 INFO:teuthology.orchestra.run.smithi142.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:03.259 INFO:teuthology.orchestra.run.smithi155.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:23:03.264 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:03.269 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:23:03.269 INFO:teuthology.orchestra.run.smithi155.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:23:03.446 INFO:teuthology.orchestra.run.smithi142.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:23:03.450 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:23:03.450 INFO:teuthology.orchestra.run.smithi142.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:23:03.506 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package upower. 2022-12-03T01:23:03.520 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:23:03.575 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:23:03.708 INFO:teuthology.orchestra.run.smithi155.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:23:03.864 INFO:teuthology.orchestra.run.smithi142.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:23:03.919 INFO:teuthology.orchestra.run.smithi155.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:03.960 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:23:03.962 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:03.975 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:23:04.036 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:04.082 INFO:teuthology.orchestra.run.smithi142.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:04.152 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:04.271 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:04.271 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:04.308 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:23:04.459 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:04.460 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:04.473 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:23:04.574 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:23:04.676 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:23:04.725 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:04.776 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:23:04.914 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:04.919 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:23:05.053 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:23:05.481 INFO:teuthology.orchestra.run.smithi137.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:05.530 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:23:05.734 INFO:teuthology.orchestra.run.smithi137.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:23:05.743 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:23:05.743 INFO:teuthology.orchestra.run.smithi137.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:23:06.120 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:06.188 INFO:teuthology.orchestra.run.smithi137.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:23:06.340 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:06.402 INFO:teuthology.orchestra.run.smithi137.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:23:06.435 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:23:06.734 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:06.734 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:23:07.190 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:07.593 INFO:teuthology.orchestra.run.smithi155.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:07.593 INFO:teuthology.orchestra.run.smithi155.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:07.661 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:07.661 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:07.738 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:07.783 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:08.625 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:09.115 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:23:09.168 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:23:09.267 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:23:09.293 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:23:10.172 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:10.173 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:10.577 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:11.978 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:23:12.078 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:23:12.900 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:23:13.572 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:23:14.001 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:23:14.101 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:14.162 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:23:14.162 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:14.182 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:23:14.266 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:14.329 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:23:14.329 INFO:teuthology.orchestra.run.smithi155.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:16.369 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:23:16.952 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:23:17.052 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:23:17.129 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:23:17.129 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:39.011 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:23:39.012 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:23:39.033 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:39.041 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ... 2022-12-03T01:23:39.065 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:23:39.066 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:23:39.080 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:39.093 INFO:teuthology.orchestra.run.smithi155.stdout:Generating grub configuration file ... 2022-12-03T01:23:39.318 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:39.338 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:39.389 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:39.411 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:39.705 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:39.716 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:39.809 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:39.821 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:41.392 INFO:teuthology.orchestra.run.smithi142.stdout:done 2022-12-03T01:23:41.743 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:41.768 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:41.769 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2022-12-03T01:23:41.769 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2022-12-03T01:23:41.770 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2022-12-03T01:23:41.770 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2022-12-03T01:23:41.770 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:41.771 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2022-12-03T01:23:41.771 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:41.771 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:41.772 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:41.773 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:41.773 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2022-12-03T01:23:41.774 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:41.774 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:41.774 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2022-12-03T01:23:41.774 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:23:41.775 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:23:41.775 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:23:41.775 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-12-03T01:23:41.776 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:23:41.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2022-12-03T01:23:41.854 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2022-12-03T01:23:42.006 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:23:42.115 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:23:42.165 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:23:42.165 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:23:42.169 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:42.181 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2022-12-03T01:23:42.195 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:23:42.229 INFO:teuthology.orchestra.run.smithi155.stdout:done 2022-12-03T01:23:42.277 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:23:42.564 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:42.587 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:42.645 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:42.678 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:42.678 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2022-12-03T01:23:42.679 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2022-12-03T01:23:42.679 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2022-12-03T01:23:42.680 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 20 2022-12-03T01:23:42.680 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:42.681 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2022-12-03T01:23:42.681 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:42.681 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:42.682 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:42.682 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:42.683 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2022-12-03T01:23:42.683 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:42.683 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:42.684 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2022-12-03T01:23:42.685 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:23:42.685 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:23:42.686 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:23:42.686 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-12-03T01:23:42.686 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:23:42.687 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2022-12-03T01:23:42.755 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2022-12-03T01:23:42.887 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:23:42.893 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:23:42.911 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:23:42.967 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:42.979 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:43.003 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:23:43.617 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:23:43.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:23:43.709 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:23:43.892 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:23:43.893 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:23:44.097 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:23:44.097 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:23:44.099 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:44.120 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:44.120 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2022-12-03T01:23:44.120 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2022-12-03T01:23:44.121 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2022-12-03T01:23:44.121 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2022-12-03T01:23:44.121 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:44.122 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2022-12-03T01:23:44.122 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:44.122 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:44.122 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:44.123 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:44.123 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2022-12-03T01:23:44.123 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:44.124 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2022-12-03T01:23:44.124 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2022-12-03T01:23:44.132 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.WZevp9rROg 2022-12-03T01:23:44.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.WZevp9rROg 2022-12-03T01:23:44.193 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.WZevp9rROg 2022-12-03T01:23:44.278 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.WZevp9rROg is 9KB 2022-12-03T01:23:44.297 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.WZevp9rROg 2022-12-03T01:23:44.305 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:44.362 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:23:44.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:23:44.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:44.396 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:23:44.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub 2022-12-03T01:23:44.527 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:23:44.550 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:23:44.619 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:23:44.804 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:23:44.805 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:23:44.921 INFO:teuthology.orchestra.run.smithi137.stdout:done 2022-12-03T01:23:45.006 INFO:teuthology.orchestra.run.smithi155.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:23:45.007 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:23:45.009 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:45.027 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:45.027 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2022-12-03T01:23:45.027 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2022-12-03T01:23:45.028 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2022-12-03T01:23:45.028 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 20 2022-12-03T01:23:45.028 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:45.028 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2022-12-03T01:23:45.029 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:45.029 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:45.029 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:45.029 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:45.029 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2022-12-03T01:23:45.030 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:45.030 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:45.030 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2022-12-03T01:23:45.031 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2022-12-03T01:23:45.076 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.auqAxUAdCV 2022-12-03T01:23:45.077 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub/grub.cfg /tmp/tmp.auqAxUAdCV 2022-12-03T01:23:45.089 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.auqAxUAdCV 2022-12-03T01:23:45.123 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.auqAxUAdCV is 9KB 2022-12-03T01:23:45.139 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.auqAxUAdCV 2022-12-03T01:23:45.146 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:45.204 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:23:45.205 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:23:45.205 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:45.241 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:23:45.242 DEBUG:teuthology.orchestra.run.smithi155:> sudo update-grub 2022-12-03T01:23:45.274 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:45.312 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:45.313 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2022-12-03T01:23:45.313 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2022-12-03T01:23:45.313 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2022-12-03T01:23:45.313 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 20 2022-12-03T01:23:45.314 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:45.314 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2022-12-03T01:23:45.314 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:45.314 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:45.315 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:45.315 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:45.315 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2022-12-03T01:23:45.316 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2022-12-03T01:23:45.317 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:23:45.317 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:23:45.317 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:23:45.317 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-12-03T01:23:45.318 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:23:45.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2022-12-03T01:23:45.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2022-12-03T01:23:45.536 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:23:45.541 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:23:45.558 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:23:45.561 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:23:45.563 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:45.575 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2022-12-03T01:23:45.651 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:23:45.810 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:45.823 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:46.190 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:46.527 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:23:46.528 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:46.538 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2022-12-03T01:23:46.751 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:46.761 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:46.946 INFO:teuthology.orchestra.run.smithi142.stderr:done 2022-12-03T01:23:46.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2022-12-03T01:23:47.095 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:47.108 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:47.157 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:23:47.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:23:47.254 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:23:47.438 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:23:47.439 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:23:47.633 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:23:47.633 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:23:47.635 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:23:47.656 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:23:47.656 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2022-12-03T01:23:47.657 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2022-12-03T01:23:47.657 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2022-12-03T01:23:47.658 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 20 2022-12-03T01:23:47.658 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:23:47.658 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2022-12-03T01:23:47.659 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:23:47.659 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:23:47.660 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:23:47.661 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:23:47.662 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2022-12-03T01:23:47.663 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2022-12-03T01:23:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:23:47.664 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2022-12-03T01:23:47.664 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2022-12-03T01:23:47.703 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.tLgqHdc9uC 2022-12-03T01:23:47.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tLgqHdc9uC 2022-12-03T01:23:47.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.tLgqHdc9uC 2022-12-03T01:23:47.804 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.tLgqHdc9uC is 9KB 2022-12-03T01:23:47.856 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.tLgqHdc9uC 2022-12-03T01:23:47.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:47.921 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:23:47.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:23:47.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:23:47.959 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:23:47.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo update-grub 2022-12-03T01:23:48.492 INFO:teuthology.orchestra.run.smithi155.stderr:done 2022-12-03T01:23:48.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2022-12-03T01:23:49.194 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:23:49.196 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:23:49.201 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2022-12-03T01:23:49.506 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:23:49.517 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:23:49.933 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:23:49.945 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:23:50.889 INFO:teuthology.orchestra.run.smithi137.stderr:done 2022-12-03T01:23:50.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2022-12-03T01:24:16.975 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-12-03T01:24:16.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-12-03T01:24:16.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:24:18.499 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-12-03T01:24:18.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-12-03T01:24:18.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:24:20.895 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-12-03T01:24:20.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-12-03T01:24:20.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:24:53.184 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2022-12-03T01:25:17.003 DEBUG:teuthology.orchestra.remote:timed out 2022-12-03T01:25:20.902 DEBUG:teuthology.orchestra.remote:timed out 2022-12-03T01:25:23.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-12-03T01:25:23.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:30.898 DEBUG:teuthology.orchestra.remote:No existing session 2022-12-03T01:25:47.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-12-03T01:25:47.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:47.309 DEBUG:teuthology.orchestra.run.smithi142:> true 2022-12-03T01:25:47.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-12-03T01:25:47.749 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-56-generic"... 2022-12-03T01:25:47.750 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2022-12-03T01:25:47.799 INFO:teuthology.orchestra.run.smithi142.stdout:5.15.0-56-generic 2022-12-03T01:25:47.800 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:25:47.800 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:25:47.800 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:25:48.801 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-12-03T01:25:48.802 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:25:48.860 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2022-12-03T01:25:48.876 DEBUG:teuthology.parallel:result is None 2022-12-03T01:25:50.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-12-03T01:25:50.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:51.194 DEBUG:teuthology.orchestra.run.smithi137:> true 2022-12-03T01:25:51.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-12-03T01:25:51.686 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-56-generic"... 2022-12-03T01:25:51.686 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2022-12-03T01:25:51.736 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-56-generic 2022-12-03T01:25:51.736 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:25:51.737 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:25:51.737 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:25:52.738 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-12-03T01:25:52.739 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:25:52.799 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2022-12-03T01:25:52.816 DEBUG:teuthology.parallel:result is None 2022-12-03T01:26:00.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-12-03T01:26:00.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:26:00.998 DEBUG:teuthology.orchestra.run.smithi155:> true 2022-12-03T01:26:01.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-12-03T01:26:01.498 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.15.0-56-generic"... 2022-12-03T01:26:01.499 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2022-12-03T01:26:01.544 INFO:teuthology.orchestra.run.smithi155.stdout:5.15.0-56-generic 2022-12-03T01:26:01.545 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:26:01.545 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:26:01.545 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:26:02.546 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-12-03T01:26:02.547 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:26:02.888 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2022-12-03T01:26:02.917 DEBUG:teuthology.parallel:result is None 2022-12-03T01:26:02.918 INFO:teuthology.run_tasks:Running task internal.base... 2022-12-03T01:26:02.926 INFO:teuthology.task.internal:Creating test directory... 2022-12-03T01:26:02.927 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:26:02.930 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:26:02.933 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:26:02.940 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-12-03T01:26:02.976 INFO:teuthology.run_tasks:Running task internal.archive... 2022-12-03T01:26:03.026 INFO:teuthology.task.internal:Creating archive directory... 2022-12-03T01:26:03.026 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:26:03.029 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:26:03.032 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:26:03.259 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-12-03T01:26:03.270 INFO:teuthology.task.internal:Enabling coredump saving... 2022-12-03T01:26:03.271 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:26:03.275 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:26:03.279 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:26:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.306 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.307 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.315 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.318 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.328 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:26:03.330 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-12-03T01:26:03.342 INFO:teuthology.task.internal:Configuring sudo... 2022-12-03T01:26:03.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:26:03.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:26:03.363 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:26:03.387 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-12-03T01:26:03.399 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-12-03T01:26:03.400 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:26:03.416 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:26:03.422 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:26:03.437 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:26:03.472 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:26:03.520 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:26:03.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:26:03.585 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:26:03.593 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:26:03.644 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:26:03.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:26:03.706 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:26:03.722 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:26:03.770 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:26:03.770 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:26:03.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2022-12-03T01:26:03.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2022-12-03T01:26:03.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2022-12-03T01:26:03.929 INFO:teuthology.run_tasks:Running task internal.timer... 2022-12-03T01:26:03.936 INFO:teuthology.task.internal:Starting timer... 2022-12-03T01:26:03.937 INFO:teuthology.run_tasks:Running task pcp... 2022-12-03T01:26:04.003 INFO:teuthology.run_tasks:Running task selinux... 2022-12-03T01:26:04.047 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2022-12-03T01:26:04.047 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux 2022-12-03T01:26:04.048 DEBUG:teuthology.task.selinux:Excluding smithi155: OS 'ubuntu' does not support SELinux 2022-12-03T01:26:04.048 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-12-03T01:26:04.048 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-12-03T01:26:04.049 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-12-03T01:26:04.049 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-12-03T01:26:04.097 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-12-03T01:26:04.097 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2022-12-03T01:26:04.121 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-12-03T01:26:04.123 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com,smithi142.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2022-12-03T01:30:37.654 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2022-12-03T01:30:37.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-12-03T01:30:37.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:30:37.786 DEBUG:teuthology.orchestra.run.smithi137:> true 2022-12-03T01:30:37.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2022-12-03T01:30:37.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:30:38.006 DEBUG:teuthology.orchestra.run.smithi142:> true 2022-12-03T01:30:38.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2022-12-03T01:30:38.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:30:38.272 DEBUG:teuthology.orchestra.run.smithi155:> true 2022-12-03T01:30:38.389 INFO:teuthology.run_tasks:Running task clock... 2022-12-03T01:30:38.400 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-12-03T01:30:38.401 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:30:38.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:30:38.406 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:30:38.407 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:30:38.410 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:30:38.410 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:30:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:30:38.436 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Command line: ntpd -gq 2022-12-03T01:30:38.437 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: proto: precision = 0.070 usec (-24) 2022-12-03T01:30:38.437 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.438 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.439 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.440 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:30:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:30:38.441 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:30:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2022-12-03T01:30:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen normally on 4 lo [::1]:123 2022-12-03T01:30:38.442 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2022-12-03T01:30:38.443 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:38 ntpd[17166]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:30:38.444 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:30:38.444 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Command line: ntpd -gq 2022-12-03T01:30:38.444 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: proto: precision = 0.060 usec (-24) 2022-12-03T01:30:38.445 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.446 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.446 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.447 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.448 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:30:38.449 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:30:38.449 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:30:38.449 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen normally on 3 enp3s0f1 172.21.15.142:123 2022-12-03T01:30:38.449 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen normally on 4 lo [::1]:123 2022-12-03T01:30:38.450 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123 2022-12-03T01:30:38.450 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:38 ntpd[17152]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:30:38.462 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:30:38.462 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Command line: ntpd -gq 2022-12-03T01:30:38.463 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: proto: precision = 0.074 usec (-24) 2022-12-03T01:30:38.463 INFO:teuthology.orchestra.run.smithi155.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.464 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:30:38.464 INFO:teuthology.orchestra.run.smithi155.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.465 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:30:38.466 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:30:38.466 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:30:38.466 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:30:38.466 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen normally on 3 enp3s0f1 172.21.15.155:123 2022-12-03T01:30:38.467 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen normally on 4 lo [::1]:123 2022-12-03T01:30:38.467 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6ded%5]:123 2022-12-03T01:30:38.467 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:38 ntpd[17155]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:30:39.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:39.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:39.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:39.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:39 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:39.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:39.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:39.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:39.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:39 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:41.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:41.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:41.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:41.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:41 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:41.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:41.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:41.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:41.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:41.467 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:41.467 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:41 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:42.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.442 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.442 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:42.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:42.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:42.445 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:42.445 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.445 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.445 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:42.446 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:42 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:42.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:42.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:42.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:42.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:42 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:43.436 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:43 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.436 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:43 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:43.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:43.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.443 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:43.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:43.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:43.445 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:43 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:43.463 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:43.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:43.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:43.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:43.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:43.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:43 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:44.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:44.442 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:44.442 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:44.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:44.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:44.445 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:44 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:44.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:44.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:44.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:44.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:44 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:45.440 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.440 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.441 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:45.442 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:45.443 INFO:teuthology.orchestra.run.smithi142.stdout: 3 Dec 01:30:45 ntpd[17152]: ntpd: time slew -0.002340 s 2022-12-03T01:30:45.443 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew -0.002340s 2022-12-03T01:30:45.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:30:45.444 INFO:teuthology.orchestra.run.smithi142.stderr: 3 Dec 01:30:45 ntpd[17152]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:30:45.464 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.465 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:45.466 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:45.467 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:45.467 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:45.467 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:45.468 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:45.469 INFO:teuthology.orchestra.run.smithi155.stdout: 3 Dec 01:30:45 ntpd[17155]: ntpd: time slew -0.002446 s 2022-12-03T01:30:45.469 INFO:teuthology.orchestra.run.smithi155.stdout:ntpd: time slew -0.002446s 2022-12-03T01:30:45.470 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:30:45.470 INFO:teuthology.orchestra.run.smithi155.stderr: 3 Dec 01:30:45 ntpd[17155]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:30:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:30:45.504 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2022-12-03T01:30:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.526 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:30:45.527 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2022-12-03T01:30:45.527 INFO:teuthology.orchestra.run.smithi155.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.527 INFO:teuthology.orchestra.run.smithi155.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.528 INFO:teuthology.orchestra.run.smithi155.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:45.528 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:46.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:46.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:46.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:46.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:46.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:46.442 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:46.443 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:46.444 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:46 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:48.437 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:48.437 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:48.437 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:48.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:48.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:48.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:48.438 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:30:48.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:30:48.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:30:48.439 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:30:48.440 INFO:teuthology.orchestra.run.smithi137.stdout: 3 Dec 01:30:48 ntpd[17166]: ntpd: time slew -0.002189 s 2022-12-03T01:30:48.440 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time slew -0.002189s 2022-12-03T01:30:48.440 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:30:48.441 INFO:teuthology.orchestra.run.smithi137.stderr: 3 Dec 01:30:48 ntpd[17166]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:30:48.576 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:30:48.576 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2022-12-03T01:30:48.576 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:48.576 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:48.577 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:48.577 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:30:48.577 INFO:teuthology.run_tasks:Running task kubeadm... 2022-12-03T01:30:48.593 INFO:tasks.kubeadm:Kubeadm start 2022-12-03T01:30:48.594 INFO:tasks.kubeadm:Config: {'pod_network': 'calico', 'version': '1.21'} 2022-12-03T01:30:48.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:30:48.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:30:48.622 DEBUG:teuthology.orchestra.run.smithi155:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:30:48.644 INFO:teuthology.orchestra.run.smithi137.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:30:48.648 INFO:teuthology.orchestra.run.smithi142.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:30:48.649 INFO:teuthology.orchestra.run.smithi155.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:30:48.652 INFO:teuthology.orchestra.run.smithi137.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:30:48.659 INFO:teuthology.orchestra.run.smithi137.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:30:48.660 INFO:teuthology.orchestra.run.smithi155.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:30:48.660 INFO:teuthology.orchestra.run.smithi142.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:30:48.665 INFO:teuthology.orchestra.run.smithi155.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:30:48.668 INFO:teuthology.orchestra.run.smithi142.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:30:48.732 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:30:48.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:30:48.789 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:30:48.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:30:48.852 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:30:48.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:30:48.869 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:30:48.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:30:48.927 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:30:48.927 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:30:48.943 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:30:48.944 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:30:48.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl restart docker || true 2022-12-03T01:30:48.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl restart docker || true 2022-12-03T01:30:49.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl restart docker || true 2022-12-03T01:30:49.987 INFO:tasks.kubeadm:Identifying pod subnet 2022-12-03T01:30:49.993 INFO:tasks.kubeadm:172.21.15.137 is in 172.21.0.0/20 at pos 3976 2022-12-03T01:30:50.070 INFO:tasks.kubeadm:Pod subnet: 10.252.64.0/21 2022-12-03T01:30:50.070 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-12-03T01:30:50.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:30:50.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:30:50.076 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:30:50.096 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:50.096 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:50.096 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:50.097 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:50.097 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:50.097 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:50.100 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:50.101 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:50.209 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:50.216 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:50.233 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:50.325 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:50.329 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:50.329 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:50.332 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:50.332 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:50.400 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:50.402 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:50.480 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:50.482 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:51.771 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:30:51.894 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:30:51.901 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:30:51.953 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:30:51.954 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:30:52.049 INFO:teuthology.orchestra.run.smithi142.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:52.063 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:52.063 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:52.064 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:52.081 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:30:52.083 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:30:52.084 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:30:52.084 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:30:52.117 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:30:52.175 INFO:teuthology.orchestra.run.smithi155.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:52.186 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:52.187 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:52.187 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:52.241 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:30:52.259 INFO:teuthology.orchestra.run.smithi137.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:52.273 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:52.273 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:52.274 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:52.312 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:30:52.313 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:30:52.326 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:30:52.426 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:30:52.427 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:30:52.502 INFO:teuthology.orchestra.run.smithi142.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:30:52.503 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-12-03T01:30:52.504 INFO:teuthology.orchestra.run.smithi142.stdout: apt-transport-https 2022-12-03T01:30:52.505 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2022-12-03T01:30:52.506 INFO:teuthology.orchestra.run.smithi142.stdout: curl libcurl4 2022-12-03T01:30:52.512 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:30:52.512 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:30:52.627 INFO:teuthology.orchestra.run.smithi155.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:30:52.628 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-12-03T01:30:52.628 INFO:teuthology.orchestra.run.smithi155.stdout: apt-transport-https 2022-12-03T01:30:52.629 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages will be upgraded: 2022-12-03T01:30:52.630 INFO:teuthology.orchestra.run.smithi155.stdout: curl libcurl4 2022-12-03T01:30:52.684 INFO:teuthology.orchestra.run.smithi155.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:52.684 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 398 kB of archives. 2022-12-03T01:30:52.685 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:30:52.685 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:30:52.686 INFO:teuthology.orchestra.run.smithi142.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:52.687 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 398 kB of archives. 2022-12-03T01:30:52.687 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:30:52.688 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:30:52.701 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:30:52.710 INFO:teuthology.orchestra.run.smithi137.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:30:52.711 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-12-03T01:30:52.711 INFO:teuthology.orchestra.run.smithi137.stdout: apt-transport-https 2022-12-03T01:30:52.712 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2022-12-03T01:30:52.713 INFO:teuthology.orchestra.run.smithi137.stdout: curl libcurl4 2022-12-03T01:30:52.763 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:30:52.766 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:30:52.767 INFO:teuthology.orchestra.run.smithi137.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:52.768 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 398 kB of archives. 2022-12-03T01:30:52.768 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:30:52.769 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:30:52.788 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:30:52.869 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:30:53.032 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:30:53.032 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:30:53.032 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:30:53.038 INFO:teuthology.orchestra.run.smithi155.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:30:53.069 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:30:53.088 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 398 kB in 0s (2,650 kB/s) 2022-12-03T01:30:53.133 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:30:53.134 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:30:53.134 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:30:53.158 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:30:53.159 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:30:53.159 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:30:53.164 INFO:teuthology.orchestra.run.smithi137.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:30:53.170 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:30:53.212 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:30:53.213 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 398 kB in 0s (2,229 kB/s) 2022-12-03T01:30:53.215 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:30:53.253 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:30:53.303 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:30:53.347 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:30:53.350 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:30:53.393 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:30:53.413 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:30:53.413 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:30:53.413 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:30:53.435 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:30:53.436 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:30:53.436 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:30:53.441 INFO:teuthology.orchestra.run.smithi142.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:30:53.493 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 398 kB in 1s (598 kB/s) 2022-12-03T01:30:53.511 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:53.586 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:30:53.626 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:30:53.629 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:30:53.639 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:53.652 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:53.668 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:30:53.779 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:53.951 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:54.079 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:54.090 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:54.247 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:54.270 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:54.418 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:54.563 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:30:54.609 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:30:54.733 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:54.743 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:30:54.766 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:30:54.859 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:54.889 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:54.994 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:30:55.015 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:55.100 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:30:55.142 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:30:55.247 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:55.373 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:30:55.439 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:30:55.499 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:30:55.586 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:30:55.926 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:30:56.052 INFO:teuthology.orchestra.run.smithi155.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:30:56.067 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:56.067 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:56.067 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:56.169 INFO:teuthology.orchestra.run.smithi137.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:30:56.183 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:56.183 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:56.183 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:56.190 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:56.191 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:56.196 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:56.213 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:56.307 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:56.320 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:56.348 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:56.419 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:56.506 INFO:teuthology.orchestra.run.smithi142.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:30:56.521 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:56.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:56.545 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:30:56.654 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:30:56.692 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:30:56.759 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:30:56.839 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:30:56.863 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:30:56.926 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:30:56.948 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:30:56.970 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:30:57.049 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 70.8 kB in 1s (85.1 kB/s) 2022-12-03T01:30:57.129 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 70.8 kB in 1s (88.2 kB/s) 2022-12-03T01:30:57.323 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:30:57.513 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 70.8 kB in 1s (83.0 kB/s) 2022-12-03T01:30:58.002 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:30:58.075 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:30:58.186 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:30:58.187 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:30:58.259 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:30:58.260 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:30:58.280 INFO:teuthology.orchestra.run.smithi155.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:58.294 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:58.294 INFO:teuthology.orchestra.run.smithi155.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:58.295 INFO:teuthology.orchestra.run.smithi155.stderr: 2022-12-03T01:30:58.351 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2022-12-03T01:30:58.356 INFO:teuthology.orchestra.run.smithi137.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:58.371 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:58.372 INFO:teuthology.orchestra.run.smithi137.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:58.372 INFO:teuthology.orchestra.run.smithi137.stderr: 2022-12-03T01:30:58.424 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-12-03T01:30:58.456 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:30:58.538 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2022-12-03T01:30:58.538 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2022-12-03T01:30:58.608 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-12-03T01:30:58.609 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-12-03T01:30:58.635 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:30:58.636 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:30:58.697 INFO:teuthology.orchestra.run.smithi155.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:30:58.697 INFO:teuthology.orchestra.run.smithi155.stdout:bridge-utils set to manually installed. 2022-12-03T01:30:58.697 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2022-12-03T01:30:58.698 INFO:teuthology.orchestra.run.smithi155.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:30:58.698 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2022-12-03T01:30:58.699 INFO:teuthology.orchestra.run.smithi155.stdout: nftables 2022-12-03T01:30:58.729 INFO:teuthology.orchestra.run.smithi142.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:30:58.740 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:58.741 INFO:teuthology.orchestra.run.smithi142.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:30:58.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2022-12-03T01:30:58.746 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2022-12-03T01:30:58.748 INFO:teuthology.orchestra.run.smithi155.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:30:58.775 INFO:teuthology.orchestra.run.smithi137.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:30:58.775 INFO:teuthology.orchestra.run.smithi137.stdout:bridge-utils set to manually installed. 2022-12-03T01:30:58.775 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2022-12-03T01:30:58.776 INFO:teuthology.orchestra.run.smithi137.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:30:58.776 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2022-12-03T01:30:58.777 INFO:teuthology.orchestra.run.smithi137.stdout: nftables 2022-12-03T01:30:58.797 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2022-12-03T01:30:58.811 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:58.811 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:30:58.811 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:30:58.812 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:30:58.830 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-12-03T01:30:58.831 INFO:teuthology.orchestra.run.smithi137.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:30:58.856 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:30:58.888 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:30:58.900 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:58.900 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:30:58.901 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:30:58.901 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:30:58.961 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:30:58.965 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2022-12-03T01:30:58.965 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2022-12-03T01:30:58.991 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:30:59.045 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:30:59.113 INFO:teuthology.orchestra.run.smithi142.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:30:59.113 INFO:teuthology.orchestra.run.smithi142.stdout:bridge-utils set to manually installed. 2022-12-03T01:30:59.114 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2022-12-03T01:30:59.115 INFO:teuthology.orchestra.run.smithi142.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:30:59.115 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2022-12-03T01:30:59.115 INFO:teuthology.orchestra.run.smithi142.stdout: nftables 2022-12-03T01:30:59.121 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:30:59.166 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2022-12-03T01:30:59.167 INFO:teuthology.orchestra.run.smithi142.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:30:59.357 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:30:59.357 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:30:59.358 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:30:59.358 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:30:59.535 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:30:59.557 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:30:59.706 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:30:59.712 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:30:59.798 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:31:00.464 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:31:00.524 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:31:00.529 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:31:01.086 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:31:01.174 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:31:01.275 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:31:01.320 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:31:01.392 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:31:01.776 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:31:01.776 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:31:01.776 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:31:01.784 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:31:01.784 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:31:01.785 INFO:teuthology.orchestra.run.smithi137.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:31:01.789 INFO:teuthology.orchestra.run.smithi137.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:31:01.799 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:31:01.841 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 81.6 MB in 3s (30.1 MB/s) 2022-12-03T01:31:01.857 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:31:01.857 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:31:01.858 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:31:01.865 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:31:01.866 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:31:01.866 INFO:teuthology.orchestra.run.smithi155.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:31:01.871 INFO:teuthology.orchestra.run.smithi155.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:31:01.919 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:31:01.923 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 81.6 MB in 3s (28.4 MB/s) 2022-12-03T01:31:01.940 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:31:01.984 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:31:01.987 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:31:02.010 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:31:02.030 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:31:02.054 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:31:02.057 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:31:02.100 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:31:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:31:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:31:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:31:02.314 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:31:02.314 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:31:02.315 INFO:teuthology.orchestra.run.smithi142.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:31:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:31:02.333 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:31:02.348 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:31:02.353 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:31:02.366 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:31:02.372 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 81.6 MB in 3s (28.1 MB/s) 2022-12-03T01:31:02.391 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:31:02.395 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:31:02.489 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:31:02.534 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:31:02.537 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:31:02.604 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:31:02.882 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:31:02.895 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:31:02.940 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:31:03.894 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:31:03.908 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:31:03.964 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:31:03.967 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:31:03.978 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:31:04.047 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:31:04.202 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:31:04.212 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:31:04.247 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:04.309 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:31:04.324 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:31:04.385 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:04.401 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:31:04.409 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:31:04.466 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:31:04.737 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:31:04.751 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:31:04.795 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:05.683 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2022-12-03T01:31:05.696 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:31:05.732 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:31:05.787 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package socat. 2022-12-03T01:31:05.800 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:31:05.829 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:31:06.078 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:31:06.091 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:31:06.119 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:31:06.199 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:31:06.218 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:31:06.231 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat. 2022-12-03T01:31:06.245 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:31:06.257 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:31:06.289 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:31:06.709 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:31:06.723 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:31:06.767 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:31:08.372 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:31:08.385 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:31:08.421 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:31:08.560 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:31:08.575 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:31:08.618 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:31:09.053 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:31:09.065 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:31:09.102 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:31:09.624 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:31:09.639 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:31:09.682 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:31:09.837 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:31:09.850 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:31:09.887 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:31:10.322 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:31:10.334 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:31:10.371 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:31:10.871 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:31:10.994 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:31:11.050 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:31:11.120 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:31:11.174 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:31:11.317 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:31:11.511 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:31:11.633 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:31:11.784 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:31:11.881 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:31:12.024 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:31:12.104 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:31:12.150 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:12.213 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:31:12.276 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:31:12.331 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:12.457 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:31:12.495 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:31:12.502 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:31:12.621 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:31:12.681 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:31:12.755 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:31:12.889 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:31:12.960 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:31:13.128 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:31:13.175 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:31:13.354 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:31:13.544 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:31:13.607 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:31:13.967 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:31:14.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:31:14.847 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:31:14.887 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:31:28.565 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:31:28.979 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:31:29.181 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:31:32.759 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:31:33.320 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:31:33.840 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:31:36.056 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:31:36.660 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:31:37.047 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:31:39.613 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:31:40.392 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:31:40.637 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:31:41.821 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:31:42.592 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:31:42.837 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:31:51.419 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:31:52.527 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:31:52.998 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:31:55.903 INFO:teuthology.orchestra.run.smithi137.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:31:56.119 INFO:teuthology.orchestra.run.smithi142.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:31:56.531 INFO:teuthology.orchestra.run.smithi155.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:31:56.534 INFO:tasks.kubeadm:Token: abcdef.kdw1m2rckqdui1pp 2022-12-03T01:31:56.535 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): '172.21.15.137', Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): '172.21.15.142', Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): '172.21.15.155'} 2022-12-03T01:31:56.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo kubeadm init --node-name smithi137 --token abcdef.kdw1m2rckqdui1pp --pod-network-cidr 10.252.64.0/21 2022-12-03T01:31:56.802 INFO:teuthology.orchestra.run.smithi137.stdout:[init] Using Kubernetes version: v1.25.4 2022-12-03T01:31:56.802 INFO:teuthology.orchestra.run.smithi137.stdout:[preflight] Running pre-flight checks 2022-12-03T01:31:57.008 INFO:teuthology.orchestra.run.smithi137.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-12-03T01:31:57.008 INFO:teuthology.orchestra.run.smithi137.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-12-03T01:31:57.009 INFO:teuthology.orchestra.run.smithi137.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-12-03T01:31:57.194 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-12-03T01:31:57.342 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "ca" certificate and key 2022-12-03T01:31:57.475 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "apiserver" certificate and key 2022-12-03T01:31:57.476 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi137] and IPs [10.96.0.1 172.21.15.137] 2022-12-03T01:31:57.572 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-12-03T01:31:57.714 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-12-03T01:31:57.855 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-12-03T01:31:58.111 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "etcd/ca" certificate and key 2022-12-03T01:31:58.537 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "etcd/server" certificate and key 2022-12-03T01:31:58.538 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi137] and IPs [172.21.15.137 127.0.0.1 ::1] 2022-12-03T01:31:58.659 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "etcd/peer" certificate and key 2022-12-03T01:31:58.659 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi137] and IPs [172.21.15.137 127.0.0.1 ::1] 2022-12-03T01:31:58.710 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-12-03T01:31:58.975 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-12-03T01:31:59.048 INFO:teuthology.orchestra.run.smithi137.stdout:[certs] Generating "sa" key and public key 2022-12-03T01:31:59.048 INFO:teuthology.orchestra.run.smithi137.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-12-03T01:31:59.474 INFO:teuthology.orchestra.run.smithi137.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-12-03T01:31:59.650 INFO:teuthology.orchestra.run.smithi137.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-12-03T01:31:59.811 INFO:teuthology.orchestra.run.smithi137.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-12-03T01:31:59.974 INFO:teuthology.orchestra.run.smithi137.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-12-03T01:31:59.986 INFO:teuthology.orchestra.run.smithi137.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:31:59.988 INFO:teuthology.orchestra.run.smithi137.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:31:59.988 INFO:teuthology.orchestra.run.smithi137.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:32:00.154 INFO:teuthology.orchestra.run.smithi137.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-12-03T01:32:00.154 INFO:teuthology.orchestra.run.smithi137.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-12-03T01:32:00.156 INFO:teuthology.orchestra.run.smithi137.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-12-03T01:32:00.156 INFO:teuthology.orchestra.run.smithi137.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-12-03T01:32:00.157 INFO:teuthology.orchestra.run.smithi137.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-12-03T01:32:00.160 INFO:teuthology.orchestra.run.smithi137.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-12-03T01:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout:[apiclient] All control plane components are healthy after 22.003580 seconds 2022-12-03T01:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-12-03T01:32:22.192 INFO:teuthology.orchestra.run.smithi137.stdout:[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster 2022-12-03T01:32:22.859 INFO:teuthology.orchestra.run.smithi137.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-12-03T01:32:22.859 INFO:teuthology.orchestra.run.smithi137.stdout:[mark-control-plane] Marking the node smithi137 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-12-03T01:32:22.859 INFO:teuthology.orchestra.run.smithi137.stdout:[mark-control-plane] Marking the node smithi137 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] 2022-12-03T01:32:23.405 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Using token: abcdef.kdw1m2rckqdui1pp 2022-12-03T01:32:23.406 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-12-03T01:32:23.424 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-12-03T01:32:23.491 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-12-03T01:32:23.499 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-12-03T01:32:23.521 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-12-03T01:32:23.532 INFO:teuthology.orchestra.run.smithi137.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-12-03T01:32:23.571 INFO:teuthology.orchestra.run.smithi137.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-12-03T01:32:24.050 INFO:teuthology.orchestra.run.smithi137.stdout:[addons] Applied essential addon: CoreDNS 2022-12-03T01:32:24.249 INFO:teuthology.orchestra.run.smithi137.stdout:[addons] Applied essential addon: kube-proxy 2022-12-03T01:32:24.251 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.251 INFO:teuthology.orchestra.run.smithi137.stdout:Your Kubernetes control-plane has initialized successfully! 2022-12-03T01:32:24.251 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.252 INFO:teuthology.orchestra.run.smithi137.stdout:To start using your cluster, you need to run the following as a regular user: 2022-12-03T01:32:24.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.252 INFO:teuthology.orchestra.run.smithi137.stdout: mkdir -p $HOME/.kube 2022-12-03T01:32:24.252 INFO:teuthology.orchestra.run.smithi137.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-12-03T01:32:24.253 INFO:teuthology.orchestra.run.smithi137.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-12-03T01:32:24.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.254 INFO:teuthology.orchestra.run.smithi137.stdout:Alternatively, if you are the root user, you can run: 2022-12-03T01:32:24.254 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.254 INFO:teuthology.orchestra.run.smithi137.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-12-03T01:32:24.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.255 INFO:teuthology.orchestra.run.smithi137.stdout:You should now deploy a pod network to the cluster. 2022-12-03T01:32:24.256 INFO:teuthology.orchestra.run.smithi137.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-12-03T01:32:24.256 INFO:teuthology.orchestra.run.smithi137.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-12-03T01:32:24.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.257 INFO:teuthology.orchestra.run.smithi137.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-12-03T01:32:24.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:32:24.258 INFO:teuthology.orchestra.run.smithi137.stdout:kubeadm join 172.21.15.137:6443 --token abcdef.kdw1m2rckqdui1pp \ 2022-12-03T01:32:24.258 INFO:teuthology.orchestra.run.smithi137.stdout: --discovery-token-ca-cert-hash sha256:4b103e4940390cebbab16cfcfea2352932a21a7f91862e026bec8301ded4be69 2022-12-03T01:32:24.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo kubeadm join 172.21.15.137:6443 --node-name smithi142 --token abcdef.kdw1m2rckqdui1pp --discovery-token-unsafe-skip-ca-verification 2022-12-03T01:32:24.266 DEBUG:teuthology.orchestra.run.smithi155:> sudo kubeadm join 172.21.15.137:6443 --node-name smithi155 --token abcdef.kdw1m2rckqdui1pp --discovery-token-unsafe-skip-ca-verification 2022-12-03T01:32:24.303 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] Running pre-flight checks 2022-12-03T01:32:24.304 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] Running pre-flight checks 2022-12-03T01:32:36.864 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] Reading configuration from the cluster... 2022-12-03T01:32:36.865 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:32:36.908 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:32:36.917 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:32:36.918 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:32:37.129 INFO:teuthology.orchestra.run.smithi142.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-12-03T01:32:37.659 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] Reading configuration from the cluster... 2022-12-03T01:32:37.660 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:32:37.884 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:32:37.885 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:32:37.885 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:32:38.100 INFO:teuthology.orchestra.run.smithi155.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-12-03T01:32:50.912 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:32:50.913 INFO:teuthology.orchestra.run.smithi142.stdout:This node has joined the cluster: 2022-12-03T01:32:50.913 INFO:teuthology.orchestra.run.smithi142.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-12-03T01:32:50.913 INFO:teuthology.orchestra.run.smithi142.stdout:* The Kubelet was informed of the new secure connection details. 2022-12-03T01:32:50.914 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:32:50.914 INFO:teuthology.orchestra.run.smithi142.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-12-03T01:32:50.915 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:32:51.320 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:32:51.320 INFO:teuthology.orchestra.run.smithi155.stdout:This node has joined the cluster: 2022-12-03T01:32:51.321 INFO:teuthology.orchestra.run.smithi155.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-12-03T01:32:51.321 INFO:teuthology.orchestra.run.smithi155.stdout:* The Kubelet was informed of the new secure connection details. 2022-12-03T01:32:51.321 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:32:51.321 INFO:teuthology.orchestra.run.smithi155.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-12-03T01:32:51.322 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:32:51.328 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:32:51.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-12-03T01:32:51.345 INFO:tasks.kubeadm:Setting up kubectl 2022-12-03T01:32:51.346 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:32:51.405 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:32:51.424 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:32:51.441 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:32:51.441 DEBUG:teuthology.orchestra.run.smithi137:> dd of=.kube/config 2022-12-03T01:32:51.455 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:32:51.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/root/.kube/config 2022-12-03T01:32:51.513 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:32:51.513 DEBUG:teuthology.orchestra.run.smithi142:> dd of=.kube/config 2022-12-03T01:32:51.521 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:32:51.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/root/.kube/config 2022-12-03T01:32:51.578 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:32:51.579 DEBUG:teuthology.orchestra.run.smithi155:> dd of=.kube/config 2022-12-03T01:32:51.587 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:32:51.589 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/root/.kube/config 2022-12-03T01:32:51.644 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-12-03T01:32:52.605 INFO:teuthology.orchestra.run.smithi137.stdout:namespace/tigera-operator created 2022-12-03T01:32:52.648 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created 2022-12-03T01:32:52.689 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created 2022-12-03T01:32:52.772 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created 2022-12-03T01:32:52.817 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created 2022-12-03T01:32:52.902 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created 2022-12-03T01:32:52.961 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created 2022-12-03T01:32:53.058 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created 2022-12-03T01:32:53.073 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created 2022-12-03T01:32:53.123 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created 2022-12-03T01:32:53.224 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created 2022-12-03T01:32:53.274 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created 2022-12-03T01:32:53.368 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created 2022-12-03T01:32:53.425 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created 2022-12-03T01:32:53.519 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created 2022-12-03T01:32:53.575 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created 2022-12-03T01:32:53.674 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created 2022-12-03T01:32:53.725 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created 2022-12-03T01:32:53.826 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created 2022-12-03T01:32:53.876 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created 2022-12-03T01:32:54.067 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created 2022-12-03T01:32:54.134 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created 2022-12-03T01:32:54.143 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/tigera-operator created 2022-12-03T01:32:54.178 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/tigera-operator created 2022-12-03T01:32:54.186 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created 2022-12-03T01:32:54.220 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps/tigera-operator created 2022-12-03T01:32:54.230 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f - 2022-12-03T01:32:57.205 INFO:teuthology.orchestra.run.smithi137.stdout:installation.operator.tigera.io/default created 2022-12-03T01:32:57.212 INFO:tasks.kubeadm:Scanning for scratch devices 2022-12-03T01:32:57.213 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:32:57.213 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:32:57.219 INFO:tasks.kubeadm:Scratch devices on smithi137: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:32:57.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:32:57.291 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records in 2022-12-03T01:32:57.292 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2022-12-03T01:32:57.292 INFO:teuthology.orchestra.run.smithi137.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155841 s, 673 MB/s 2022-12-03T01:32:57.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:32:57.366 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records in 2022-12-03T01:32:57.367 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2022-12-03T01:32:57.368 INFO:teuthology.orchestra.run.smithi137.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0170783 s, 614 MB/s 2022-12-03T01:32:57.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:32:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records in 2022-12-03T01:32:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2022-12-03T01:32:57.443 INFO:teuthology.orchestra.run.smithi137.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0170464 s, 615 MB/s 2022-12-03T01:32:57.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:32:57.519 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records in 2022-12-03T01:32:57.519 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2022-12-03T01:32:57.520 INFO:teuthology.orchestra.run.smithi137.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155899 s, 673 MB/s 2022-12-03T01:32:57.521 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:32:57.521 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:32:57.529 INFO:tasks.kubeadm:Scratch devices on smithi142: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:32:57.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:32:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-12-03T01:32:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-12-03T01:32:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139665 s, 751 MB/s 2022-12-03T01:32:57.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:32:57.671 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-12-03T01:32:57.672 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-12-03T01:32:57.672 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015424 s, 680 MB/s 2022-12-03T01:32:57.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:32:57.745 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-12-03T01:32:57.746 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-12-03T01:32:57.746 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0158796 s, 660 MB/s 2022-12-03T01:32:57.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:32:57.827 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records in 2022-12-03T01:32:57.828 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2022-12-03T01:32:57.828 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180215 s, 582 MB/s 2022-12-03T01:32:57.829 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:32:57.830 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:32:57.838 INFO:tasks.kubeadm:Scratch devices on smithi155: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:32:57.838 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:32:57.915 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-12-03T01:32:57.915 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-12-03T01:32:57.916 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0198828 s, 527 MB/s 2022-12-03T01:32:57.917 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:32:57.987 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-12-03T01:32:57.988 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-12-03T01:32:57.988 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140707 s, 745 MB/s 2022-12-03T01:32:57.989 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:32:58.064 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-12-03T01:32:58.065 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-12-03T01:32:58.065 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0167188 s, 627 MB/s 2022-12-03T01:32:58.067 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:32:58.141 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records in 2022-12-03T01:32:58.141 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2022-12-03T01:32:58.142 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180442 s, 581 MB/s 2022-12-03T01:32:58.161 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-12-03T01:32:58.162 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi137-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi137 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi137-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi137 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi137-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi137 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi137-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi137 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi142-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi142 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi155-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi155 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-12-03T01:32:58.162 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f - 2022-12-03T01:32:59.005 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi137-lv-1 created 2022-12-03T01:32:59.058 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi137-lv-2 created 2022-12-03T01:32:59.081 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi137-lv-3 created 2022-12-03T01:32:59.110 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi137-lv-4 created 2022-12-03T01:32:59.130 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi142-lv-1 created 2022-12-03T01:32:59.146 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi142-lv-2 created 2022-12-03T01:32:59.208 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi142-lv-3 created 2022-12-03T01:32:59.229 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi142-lv-4 created 2022-12-03T01:32:59.252 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi155-lv-1 created 2022-12-03T01:32:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi155-lv-2 created 2022-12-03T01:32:59.288 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi155-lv-3 created 2022-12-03T01:32:59.350 INFO:teuthology.orchestra.run.smithi137.stdout:persistentvolume/smithi155-lv-4 created 2022-12-03T01:32:59.371 INFO:teuthology.orchestra.run.smithi137.stdout:storageclass.storage.k8s.io/scratch created 2022-12-03T01:32:59.376 DEBUG:teuthology.orchestra.run.smithi137:> kubectl taint node smithi137 node-role.kubernetes.io/master- || true 2022-12-03T01:32:59.444 INFO:teuthology.orchestra.run.smithi137.stderr:error: taint "node-role.kubernetes.io/master" not found 2022-12-03T01:32:59.447 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-12-03T01:32:59.447 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-12-03T01:32:59.463 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-12-03T01:32:59.465 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:32:59.465 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:32:59.473 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:32:59.474 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2022-12-03T01:32:59.522 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:32:59.523 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:32:59.523 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 495 Links: 1 2022-12-03T01:32:59.523 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:32:59.523 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2022-12-03 01:32:57.291510392 +0000 2022-12-03T01:32:59.524 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2022-12-03 01:32:57.291510392 +0000 2022-12-03T01:32:59.524 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2022-12-03 01:32:57.291510392 +0000 2022-12-03T01:32:59.524 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2022-12-03T01:32:59.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:32:59.578 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2022-12-03T01:32:59.578 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2022-12-03T01:32:59.578 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000631185 s, 811 kB/s 2022-12-03T01:32:59.579 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:32:59.630 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2022-12-03T01:32:59.680 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:32:59.680 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:32:59.680 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-12-03T01:32:59.681 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:32:59.681 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2022-12-03 01:32:57.371510390 +0000 2022-12-03T01:32:59.681 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2022-12-03 01:32:57.367510390 +0000 2022-12-03T01:32:59.681 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2022-12-03 01:32:57.367510390 +0000 2022-12-03T01:32:59.682 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2022-12-03T01:32:59.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:32:59.739 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2022-12-03T01:32:59.740 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2022-12-03T01:32:59.740 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000509238 s, 1.0 MB/s 2022-12-03T01:32:59.742 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:32:59.794 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2022-12-03T01:32:59.844 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:32:59.844 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:32:59.844 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 507 Links: 1 2022-12-03T01:32:59.845 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:32:59.845 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2022-12-03 01:32:57.447510388 +0000 2022-12-03T01:32:59.845 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2022-12-03 01:32:57.447510388 +0000 2022-12-03T01:32:59.845 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2022-12-03 01:32:57.447510388 +0000 2022-12-03T01:32:59.846 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2022-12-03T01:32:59.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:32:59.899 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2022-12-03T01:32:59.900 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2022-12-03T01:32:59.900 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000553914 s, 924 kB/s 2022-12-03T01:32:59.901 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:32:59.958 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2022-12-03T01:33:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:33:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:00.008 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-12-03T01:33:00.009 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:00.009 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2022-12-03 01:32:57.523510386 +0000 2022-12-03T01:33:00.009 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2022-12-03 01:32:57.523510386 +0000 2022-12-03T01:33:00.009 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2022-12-03 01:32:57.523510386 +0000 2022-12-03T01:33:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2022-12-03T01:33:00.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:33:00.058 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2022-12-03T01:33:00.059 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2022-12-03T01:33:00.060 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00056582 s, 905 kB/s 2022-12-03T01:33:00.062 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:33:00.118 DEBUG:teuthology.orchestra.run.smithi137:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:33:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:loop 2022-12-03T01:33:00.295 INFO:tasks.nvme_loop:Connecting nvme_loop smithi137:/dev/vg_nvme/lv_1... 2022-12-03T01:33:00.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:33:00.320 INFO:teuthology.orchestra.run.smithi137.stdout:1 2022-12-03T01:33:00.343 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:33:00.367 INFO:tasks.nvme_loop:Connecting nvme_loop smithi137:/dev/vg_nvme/lv_2... 2022-12-03T01:33:00.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:33:00.445 INFO:teuthology.orchestra.run.smithi137.stdout:1 2022-12-03T01:33:00.470 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:33:00.492 INFO:tasks.nvme_loop:Connecting nvme_loop smithi137:/dev/vg_nvme/lv_3... 2022-12-03T01:33:00.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:33:00.560 INFO:teuthology.orchestra.run.smithi137.stdout:1 2022-12-03T01:33:00.585 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:33:00.609 INFO:tasks.nvme_loop:Connecting nvme_loop smithi137:/dev/vg_nvme/lv_4... 2022-12-03T01:33:00.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:33:00.673 INFO:teuthology.orchestra.run.smithi137.stdout:1 2022-12-03T01:33:00.697 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:33:00.720 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:33:00.721 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:33:00.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo nvme list 2022-12-03T01:33:00.828 INFO:teuthology.orchestra.run.smithi137.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:33:00.828 INFO:teuthology.orchestra.run.smithi137.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:33:00.828 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/nvme0n1 CVFT6233004A400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-12-03T01:33:00.829 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/nvme1n1 eaad14f061550825e267 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:00.829 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/nvme2n1 9c0538babaa34a1f979a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:00.829 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/nvme3n1 ecef3a851e20e83a0cce Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:00.829 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/nvme4n1 f48c15c4bf0b242d4ec9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:00.830 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:33:00.831 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:33:00.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/scratch_devs 2022-12-03T01:33:00.888 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:33:00.889 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:33:00.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:33:00.898 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2022-12-03T01:33:00.946 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:33:00.947 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:00.947 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 492 Links: 1 2022-12-03T01:33:00.947 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:00.948 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-12-03 01:32:57.597739164 +0000 2022-12-03T01:33:00.948 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-12-03 01:32:57.597739164 +0000 2022-12-03T01:33:00.948 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-12-03 01:32:57.597739164 +0000 2022-12-03T01:33:00.948 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-12-03T01:33:00.949 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:33:01.002 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-12-03T01:33:01.002 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-12-03T01:33:01.002 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000606029 s, 845 kB/s 2022-12-03T01:33:01.004 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:33:01.060 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2022-12-03T01:33:01.110 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:33:01.111 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:01.112 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-12-03T01:33:01.112 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:01.112 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-12-03 01:32:57.673739162 +0000 2022-12-03T01:33:01.113 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-12-03 01:32:57.673739162 +0000 2022-12-03T01:33:01.113 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-12-03 01:32:57.673739162 +0000 2022-12-03T01:33:01.113 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-12-03T01:33:01.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:33:01.168 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-12-03T01:33:01.168 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-12-03T01:33:01.169 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000584941 s, 875 kB/s 2022-12-03T01:33:01.169 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:33:01.224 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2022-12-03T01:33:01.279 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:33:01.279 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:01.280 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-12-03T01:33:01.280 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:01.280 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-12-03 01:32:57.745739161 +0000 2022-12-03T01:33:01.281 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-12-03 01:32:57.745739161 +0000 2022-12-03T01:33:01.281 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-12-03 01:32:57.745739161 +0000 2022-12-03T01:33:01.282 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-12-03T01:33:01.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:33:01.344 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-12-03T01:33:01.344 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-12-03T01:33:01.345 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000559398 s, 915 kB/s 2022-12-03T01:33:01.347 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:33:01.401 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2022-12-03T01:33:01.455 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:33:01.455 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:01.455 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-12-03T01:33:01.456 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:01.456 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2022-12-03 01:32:57.829739159 +0000 2022-12-03T01:33:01.456 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2022-12-03 01:32:57.825739159 +0000 2022-12-03T01:33:01.457 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2022-12-03 01:32:57.825739159 +0000 2022-12-03T01:33:01.457 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2022-12-03T01:33:01.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:33:01.514 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2022-12-03T01:33:01.514 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2022-12-03T01:33:01.515 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000512046 s, 1000 kB/s 2022-12-03T01:33:01.516 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:33:01.573 DEBUG:teuthology.orchestra.run.smithi142:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:33:01.800 INFO:teuthology.orchestra.run.smithi142.stdout:loop 2022-12-03T01:33:01.802 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_1... 2022-12-03T01:33:01.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:33:01.821 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-12-03T01:33:01.846 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:33:01.869 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_2... 2022-12-03T01:33:01.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:33:01.936 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-12-03T01:33:01.965 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:33:01.986 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_3... 2022-12-03T01:33:01.987 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:33:02.048 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-12-03T01:33:02.070 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:33:02.094 INFO:tasks.nvme_loop:Connecting nvme_loop smithi142:/dev/vg_nvme/lv_4... 2022-12-03T01:33:02.095 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:33:02.162 INFO:teuthology.orchestra.run.smithi142.stdout:1 2022-12-03T01:33:02.186 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:33:02.205 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:33:02.206 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:33:02.259 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme list 2022-12-03T01:33:02.315 INFO:teuthology.orchestra.run.smithi142.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:33:02.316 INFO:teuthology.orchestra.run.smithi142.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:33:02.316 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme0n1 CVFT6303009Y400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-12-03T01:33:02.316 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme1n1 ca4cb25069232146eca7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:02.317 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme2n1 f8941b509f552f7a70b5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:02.317 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme3n1 4ef527419f4dd3dcb6be Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:02.318 INFO:teuthology.orchestra.run.smithi142.stdout:/dev/nvme4n1 a960223ec227315c5480 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:02.320 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:33:02.321 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:33:02.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/scratch_devs 2022-12-03T01:33:02.377 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:33:02.378 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:33:02.385 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:33:02.386 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2022-12-03T01:33:02.436 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:33:02.437 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:02.437 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 495 Links: 1 2022-12-03T01:33:02.438 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:02.438 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-12-03 01:32:57.918390578 +0000 2022-12-03T01:33:02.439 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-12-03 01:32:57.918390578 +0000 2022-12-03T01:33:02.439 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-12-03 01:32:57.918390578 +0000 2022-12-03T01:33:02.439 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-12-03T01:33:02.440 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:33:02.495 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-12-03T01:33:02.496 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-12-03T01:33:02.496 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000604093 s, 848 kB/s 2022-12-03T01:33:02.498 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:33:02.553 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2022-12-03T01:33:02.608 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:33:02.609 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:02.609 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-12-03T01:33:02.610 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:02.610 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-12-03 01:32:57.994390577 +0000 2022-12-03T01:33:02.610 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-12-03 01:32:57.994390577 +0000 2022-12-03T01:33:02.611 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-12-03 01:32:57.994390577 +0000 2022-12-03T01:33:02.611 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-12-03T01:33:02.612 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:33:02.670 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-12-03T01:33:02.670 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-12-03T01:33:02.670 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000548388 s, 934 kB/s 2022-12-03T01:33:02.672 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:33:02.721 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2022-12-03T01:33:02.776 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:33:02.776 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:02.777 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-12-03T01:33:02.777 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:02.777 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-12-03 01:32:58.070390575 +0000 2022-12-03T01:33:02.777 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-12-03 01:32:58.070390575 +0000 2022-12-03T01:33:02.778 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-12-03 01:32:58.070390575 +0000 2022-12-03T01:33:02.778 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-12-03T01:33:02.778 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:33:02.835 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-12-03T01:33:02.835 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-12-03T01:33:02.836 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000542054 s, 945 kB/s 2022-12-03T01:33:02.837 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:33:02.889 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2022-12-03T01:33:02.947 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:33:02.947 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:33:02.948 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 515 Links: 1 2022-12-03T01:33:02.948 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:33:02.949 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2022-12-03 01:32:58.146390573 +0000 2022-12-03T01:33:02.949 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2022-12-03 01:32:58.146390573 +0000 2022-12-03T01:33:02.950 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2022-12-03 01:32:58.146390573 +0000 2022-12-03T01:33:02.950 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2022-12-03T01:33:02.951 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:33:03.004 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2022-12-03T01:33:03.005 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2022-12-03T01:33:03.005 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000555404 s, 922 kB/s 2022-12-03T01:33:03.006 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:33:03.058 DEBUG:teuthology.orchestra.run.smithi155:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:33:03.193 INFO:teuthology.orchestra.run.smithi155.stdout:loop 2022-12-03T01:33:03.194 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_1... 2022-12-03T01:33:03.194 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:33:03.220 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-12-03T01:33:03.241 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:33:03.263 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_2... 2022-12-03T01:33:03.263 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:33:03.334 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-12-03T01:33:03.367 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:33:03.387 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_3... 2022-12-03T01:33:03.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:33:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-12-03T01:33:03.481 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:33:03.506 INFO:tasks.nvme_loop:Connecting nvme_loop smithi155:/dev/vg_nvme/lv_4... 2022-12-03T01:33:03.506 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:33:03.579 INFO:teuthology.orchestra.run.smithi155.stdout:1 2022-12-03T01:33:03.601 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:33:03.623 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:33:03.624 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:33:03.671 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme list 2022-12-03T01:33:03.728 INFO:teuthology.orchestra.run.smithi155.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:33:03.728 INFO:teuthology.orchestra.run.smithi155.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:33:03.729 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme0n1 CVFT623300GW400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-12-03T01:33:03.729 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme1n1 0e0488de06488a71b0a0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:03.730 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme2n1 f92aaad69ea167b3eac8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:03.730 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme3n1 3f3499725f8114a02553 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:03.730 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/nvme4n1 9552b95b09ccb3096106 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:33:03.731 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:33:03.732 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:33:03.732 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/scratch_devs 2022-12-03T01:33:03.783 INFO:teuthology.run_tasks:Running task rook... 2022-12-03T01:33:03.887 INFO:tasks.rook:Rook start 2022-12-03T01:33:03.888 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '48d203d9b836cc224eb4085f6198d948c6fa1bb1', 'rook_image': 'rook/ceph:master'} 2022-12-03T01:33:03.889 INFO:tasks.rook:Ceph image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1 2022-12-03T01:33:03.889 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-12-03T01:33:03.890 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-12-03T01:33:03.899 INFO:teuthology.orchestra.run.smithi137.stderr:Cloning into 'rook'... 2022-12-03T01:33:07.154 INFO:tasks.rook:/home/teuthworker 2022-12-03T01:33:07.155 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/sbin:/usr/sbin ip addr show 2022-12-03T01:33:07.200 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-12-03T01:33:07.201 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-12-03T01:33:07.201 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2022-12-03T01:33:07.201 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:33:07.202 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2022-12-03T01:33:07.202 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:33:07.202 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:33:07.202 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2022-12-03T01:33:07.203 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2022-12-03T01:33:07.203 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:33:07.203 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2022-12-03T01:33:07.204 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2022-12-03T01:33:07.204 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:33:07.204 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:33:07.204 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-12-03T01:33:07.205 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:33:07.205 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-12-03T01:33:07.205 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42742sec preferred_lft 42742sec 2022-12-03T01:33:07.205 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2022-12-03T01:33:07.206 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:33:07.206 INFO:teuthology.orchestra.run.smithi137.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-12-03T01:33:07.206 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 02:42:d5:f4:da:84 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:33:07.206 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-12-03T01:33:07.207 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:33:07.208 DEBUG:teuthology.orchestra.run.smithi137:> which systemctl 2022-12-03T01:33:07.249 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/systemctl 2022-12-03T01:33:07.250 DEBUG:teuthology.orchestra.run.smithi137:> true 2022-12-03T01:33:07.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2022-12-03T01:33:07.385 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2022-12-03T01:33:07.391 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-12-03T01:33:07.401 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'is_mounted', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-12-03T01:33:07.444 DEBUG:teuthology.orchestra.remote:smithi137:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:33:07.494 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:33:07.494 DEBUG:teuthology.orchestra.run.smithi137:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-12-03T01:33:07.502 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-12-03T01:33:07.550 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:33:07.550 DEBUG:teuthology.orchestra.run.smithi137:> dd of=operator.yaml 2022-12-03T01:33:07.559 INFO:tasks.rook:Deploying operator 2022-12-03T01:33:07.610 DEBUG:teuthology.orchestra.remote:smithi137:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:33:07.715 DEBUG:teuthology.orchestra.remote:smithi137:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:33:07.731 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-12-03T01:33:08.016 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created 2022-12-03T01:33:08.068 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-12-03T01:33:08.151 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-12-03T01:33:08.210 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-12-03T01:33:08.318 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-12-03T01:33:08.554 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-12-03T01:33:08.788 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-12-03T01:33:08.876 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-12-03T01:33:08.997 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-12-03T01:33:09.285 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-12-03T01:33:09.322 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-12-03T01:33:09.442 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-12-03T01:33:09.507 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-12-03T01:33:09.624 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-12-03T01:33:09.721 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-12-03T01:33:09.857 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-12-03T01:33:09.908 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-12-03T01:33:10.049 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-12-03T01:33:10.066 INFO:teuthology.orchestra.run.smithi137.stdout:namespace/rook-ceph created 2022-12-03T01:33:10.118 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-12-03T01:33:10.244 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-12-03T01:33:10.312 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:33:10.398 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-12-03T01:33:10.434 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-12-03T01:33:10.495 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-12-03T01:33:10.540 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-12-03T01:33:10.558 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-12-03T01:33:10.585 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-12-03T01:33:10.600 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:33:10.617 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:33:10.625 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-12-03T01:33:10.633 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:33:10.701 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-12-03T01:33:10.752 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-12-03T01:33:10.788 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-12-03T01:33:10.850 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-12-03T01:33:10.869 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:33:10.912 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:33:11.046 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-12-03T01:33:11.084 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:33:11.137 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-12-03T01:33:11.215 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-12-03T01:33:11.242 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-12-03T01:33:11.293 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:33:11.388 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-12-03T01:33:11.453 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-12-03T01:33:11.484 INFO:teuthology.orchestra.run.smithi137.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:33:11.548 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-12-03T01:33:11.570 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-12-03T01:33:11.594 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-12-03T01:33:11.617 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-12-03T01:33:11.722 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-12-03T01:33:11.765 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-12-03T01:33:11.796 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-12-03T01:33:11.908 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:33:11.950 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-12-03T01:33:12.073 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-12-03T01:33:12.109 INFO:teuthology.orchestra.run.smithi137.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:33:12.142 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-12-03T01:33:12.170 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-mgr created 2022-12-03T01:33:12.230 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-osd created 2022-12-03T01:33:12.250 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-purge-osd created 2022-12-03T01:33:12.325 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-rgw created 2022-12-03T01:33:12.377 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-ceph-system created 2022-12-03T01:33:12.400 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-12-03T01:33:12.427 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-12-03T01:33:12.450 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-12-03T01:33:12.475 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-12-03T01:33:12.578 INFO:teuthology.orchestra.run.smithi137.stdout:configmap/rook-ceph-operator-config created 2022-12-03T01:33:12.630 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps/rook-ceph-operator created 2022-12-03T01:33:12.635 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:33:12.748 INFO:teuthology.orchestra.run.smithi137.stderr:No resources found in rook-ceph namespace. 2022-12-03T01:33:22.754 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:33:22.824 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:33:22.824 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 Pending 0 10s 2022-12-03T01:33:32.827 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:33:33.013 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:33:33.014 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 Pending 0 21s 2022-12-03T01:33:43.018 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:33:43.093 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:33:43.093 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 31s 2022-12-03T01:33:53.096 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:33:53.175 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:33:53.176 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 41s 2022-12-03T01:34:03.179 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:34:03.261 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:03.261 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 51s 2022-12-03T01:34:13.264 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:34:13.373 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:13.374 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 61s 2022-12-03T01:34:23.378 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:34:23.454 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:23.454 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 71s 2022-12-03T01:34:33.457 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:34:33.530 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:33.531 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 0/1 ContainerCreating 0 81s 2022-12-03T01:34:43.534 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:34:43.603 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:43.604 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 91s 2022-12-03T01:34:43.606 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph logs -f rook-ceph-operator-5ddcc89457-2gd84 2022-12-03T01:34:43.615 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:34:43.615 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:34:43.663 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:34:43.664 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:34:43.674 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:34:43.674 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:34:43.683 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-12-03T01:34:43.684 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-12-03T01:34:43.684 INFO:tasks.rook: override: [mon] debug mon = 20 2022-12-03T01:34:43.684 INFO:tasks.rook: override: [mon] debug ms = 1 2022-12-03T01:34:43.685 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-12-03T01:34:43.685 INFO:tasks.rook: override: [osd] debug ms = 1 2022-12-03T01:34:43.685 INFO:tasks.rook: override: [osd] debug osd = 20 2022-12-03T01:34:43.686 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-12-03T01:34:43.688 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f - 2022-12-03T01:34:43.701 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:35.944365 I | rookcmd: starting Rook v1.10.0-alpha.0.416.gfe1ef6c22 with arguments '/usr/local/bin/rook ceph operator' 2022-12-03T01:34:43.702 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:35.944473 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-12-03T01:34:43.702 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:35.944476 I | cephcmd: starting Rook-Ceph operator 2022-12-03T01:34:43.702 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.090924 I | cephcmd: base ceph version inside the rook operator image is "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)" 2022-12-03T01:34:43.703 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.097526 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-12-03T01:34:43.703 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.097545 I | operator: watching all namespaces for Ceph CRs 2022-12-03T01:34:43.703 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.097620 I | operator: setting up schemes 2022-12-03T01:34:43.704 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.101355 I | operator: setting up the controller-runtime manager 2022-12-03T01:34:43.704 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.110874 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-12-03T01:34:43.705 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112747 I | operator: failed to get cert manager 2022-12-03T01:34:43.705 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112777 I | ceph-cluster-controller: successfully started 2022-12-03T01:34:43.705 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112843 I | ceph-cluster-controller: enabling hotplug orchestration 2022-12-03T01:34:43.706 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112862 I | ceph-crashcollector-controller: successfully started 2022-12-03T01:34:43.706 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112883 I | ceph-block-pool-controller: successfully started 2022-12-03T01:34:43.707 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112907 I | ceph-object-store-user-controller: successfully started 2022-12-03T01:34:43.707 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112928 I | ceph-object-realm-controller: successfully started 2022-12-03T01:34:43.708 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112940 I | ceph-object-zonegroup-controller: successfully started 2022-12-03T01:34:43.708 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.112954 I | ceph-object-zone-controller: successfully started 2022-12-03T01:34:43.709 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113052 I | ceph-object-controller: successfully started 2022-12-03T01:34:43.709 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113088 I | ceph-file-controller: successfully started 2022-12-03T01:34:43.710 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113118 I | ceph-nfs-controller: successfully started 2022-12-03T01:34:43.710 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113146 I | ceph-rbd-mirror-controller: successfully started 2022-12-03T01:34:43.714 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113165 I | ceph-client-controller: successfully started 2022-12-03T01:34:43.714 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113181 I | ceph-filesystem-mirror-controller: successfully started 2022-12-03T01:34:43.715 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113203 I | operator: rook-ceph-operator-config-controller successfully started 2022-12-03T01:34:43.715 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113219 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-12-03T01:34:43.715 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113232 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-12-03T01:34:43.716 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113253 I | ceph-bucket-topic: successfully started 2022-12-03T01:34:43.716 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113269 I | ceph-bucket-notification: successfully started 2022-12-03T01:34:43.716 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113280 I | ceph-bucket-notification: successfully started 2022-12-03T01:34:43.716 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113291 I | ceph-fs-subvolumegroup-controller: successfully started 2022-12-03T01:34:43.717 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.113302 I | blockpool-rados-namespace-controller: successfully started 2022-12-03T01:34:43.717 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.114148 I | operator: starting the controller-runtime manager 2022-12-03T01:34:43.717 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.217880 I | ceph-csi: CSI Ceph RBD driver disabled 2022-12-03T01:34:43.717 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.217909 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-12-03T01:34:43.718 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.220773 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-12-03T01:34:43.718 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.220806 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-12-03T01:34:43.718 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.220822 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-12-03T01:34:43.718 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.225910 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap) 2022-12-03T01:34:43.718 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.225936 I | operator: rook-ceph-operator-config-controller done reconciling 2022-12-03T01:34:43.719 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.226133 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-12-03T01:34:43.719 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.267609 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-12-03T01:34:43.719 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.267629 I | ceph-csi: CSI CephFS driver disabled 2022-12-03T01:34:43.719 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.267637 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-12-03T01:34:43.720 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.270316 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-12-03T01:34:43.720 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.301074 I | ceph-csi: successfully removed CSI CephFS driver 2022-12-03T01:34:43.720 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.301100 I | ceph-csi: CSI NFS driver disabled 2022-12-03T01:34:43.721 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.301110 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-12-03T01:34:43.721 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.303508 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-12-03T01:34:43.721 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:36.309198 I | ceph-csi: successfully removed CSI NFS driver 2022-12-03T01:34:44.676 INFO:teuthology.orchestra.run.smithi137.stdout:configmap/rook-config-override created 2022-12-03T01:34:44.684 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true storage: storageClassDeviceSets: - count: 12 name: scratch portable: false volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: scratch volumeMode: Block 2022-12-03T01:34:44.684 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:34:44.685 DEBUG:teuthology.orchestra.run.smithi137:> dd of=cluster.yaml 2022-12-03T01:34:44.692 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f cluster.yaml 2022-12-03T01:34:44.982 INFO:teuthology.orchestra.run.smithi137.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-12-03T01:34:44.984 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:44.984390 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-12-03T01:34:44.985 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:44.985103 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:34:44.998 DEBUG:teuthology.orchestra.remote:smithi137:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:34:45.012 DEBUG:teuthology.orchestra.run.smithi137:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-12-03T01:34:45.029 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.028905 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-12-03T01:34:45.041 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.040692 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-12-03T01:34:45.041 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.040770 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-12-03T01:34:45.087 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085915 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-12-03T01:34:45.088 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085937 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-12-03T01:34:45.088 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085944 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap) 2022-12-03T01:34:45.088 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085951 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-12-03T01:34:45.089 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085957 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-12-03T01:34:45.089 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085965 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:34:45.089 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085971 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-12-03T01:34:45.089 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085978 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap) 2022-12-03T01:34:45.089 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085986 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-12-03T01:34:45.090 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085992 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-12-03T01:34:45.090 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.085999 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-12-03T01:34:45.090 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086005 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-12-03T01:34:45.090 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086012 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-12-03T01:34:45.091 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086018 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-12-03T01:34:45.091 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086026 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-12-03T01:34:45.091 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086036 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-12-03T01:34:45.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086047 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-12-03T01:34:45.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086058 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-12-03T01:34:45.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086069 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap) 2022-12-03T01:34:45.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086076 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap) 2022-12-03T01:34:45.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086084 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap) 2022-12-03T01:34:45.093 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086095 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-12-03T01:34:45.093 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086122 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-12-03T01:34:45.093 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086134 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-12-03T01:34:45.093 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086144 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap) 2022-12-03T01:34:45.094 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086155 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-12-03T01:34:45.094 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086165 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap) 2022-12-03T01:34:45.094 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086176 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap) 2022-12-03T01:34:45.094 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086188 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default) 2022-12-03T01:34:45.094 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086199 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:34:45.095 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086213 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:34:45.095 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086224 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:34:45.095 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086236 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default) 2022-12-03T01:34:45.095 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086246 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-12-03T01:34:45.096 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086257 I | ceph-csi: Kubernetes version is 1.25 2022-12-03T01:34:45.096 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086270 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.6.0" (default) 2022-12-03T01:34:45.096 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086284 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-12-03T01:34:45.096 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.086294 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default) 2022-12-03T01:34:45.098 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.087751 I | ceph-spec: detecting the ceph image version for image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1... 2022-12-03T01:34:45.099 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091099 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-12-03T01:34:45.099 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091117 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.7.2" (default) 2022-12-03T01:34:45.099 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091127 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" (default) 2022-12-03T01:34:45.099 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091137 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v3.3.0" (default) 2022-12-03T01:34:45.100 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091148 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.0.0" (default) 2022-12-03T01:34:45.100 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091158 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" (default) 2022-12-03T01:34:45.100 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091168 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-12-03T01:34:45.100 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091177 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.5.0" (default) 2022-12-03T01:34:45.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091184 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default) 2022-12-03T01:34:45.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091191 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-12-03T01:34:45.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091200 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default) 2022-12-03T01:34:45.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091212 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-12-03T01:34:45.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091220 I | op-k8sutil: CSI_CLUSTER_NAME="" (default) 2022-12-03T01:34:45.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091229 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default) 2022-12-03T01:34:45.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091238 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.7.2" 2022-12-03T01:34:45.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091325 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:34:45.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:34:45.091336 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:34:45.253 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps/rook-ceph-tools created 2022-12-03T01:34:45.260 INFO:tasks.rook:Waiting for tools container to start 2022-12-03T01:34:45.260 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:34:45.372 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:45.372 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 Pending 0 0s 2022-12-03T01:34:45.377 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:34:45.522 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:45.522 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Pending 0 0s 2022-12-03T01:34:45.523 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 0s 2022-12-03T01:34:45.523 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 93s 2022-12-03T01:34:45.523 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 Pending 0 0s 2022-12-03T01:34:50.525 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:34:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:50.610 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 5s 2022-12-03T01:34:50.612 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:34:50.694 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:50.694 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 5s 2022-12-03T01:34:50.695 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 5s 2022-12-03T01:34:50.695 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 98s 2022-12-03T01:34:50.696 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 5s 2022-12-03T01:34:55.697 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:34:55.774 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:55.774 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 10s 2022-12-03T01:34:55.776 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:34:55.849 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:34:55.850 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 10s 2022-12-03T01:34:55.850 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 10s 2022-12-03T01:34:55.850 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 103s 2022-12-03T01:34:55.850 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 10s 2022-12-03T01:35:00.853 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:00.931 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:00.932 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 15s 2022-12-03T01:35:00.933 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:01.007 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:01.008 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 16s 2022-12-03T01:35:01.008 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 16s 2022-12-03T01:35:01.008 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 109s 2022-12-03T01:35:01.009 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 16s 2022-12-03T01:35:06.012 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:06.095 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:06.095 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 21s 2022-12-03T01:35:06.097 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:06.175 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:06.176 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 21s 2022-12-03T01:35:06.176 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 21s 2022-12-03T01:35:06.176 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 114s 2022-12-03T01:35:06.177 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 21s 2022-12-03T01:35:11.179 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:11.258 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:11.259 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 26s 2022-12-03T01:35:11.261 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:11.331 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:11.331 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 26s 2022-12-03T01:35:11.331 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 26s 2022-12-03T01:35:11.332 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 119s 2022-12-03T01:35:11.332 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 26s 2022-12-03T01:35:16.334 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:16.413 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:16.414 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 31s 2022-12-03T01:35:16.415 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:16.504 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:16.505 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 31s 2022-12-03T01:35:16.505 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 31s 2022-12-03T01:35:16.506 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m4s 2022-12-03T01:35:16.506 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 31s 2022-12-03T01:35:21.509 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:21.590 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:21.590 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 36s 2022-12-03T01:35:21.592 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:21.669 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:21.669 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Init:0/1 0 36s 2022-12-03T01:35:21.670 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Init:0/1 0 36s 2022-12-03T01:35:21.670 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m9s 2022-12-03T01:35:21.670 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 36s 2022-12-03T01:35:26.673 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:26.762 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:26.762 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 41s 2022-12-03T01:35:26.764 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:26.842 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:26.842 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 41s 2022-12-03T01:35:26.843 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 PodInitializing 0 41s 2022-12-03T01:35:26.843 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m14s 2022-12-03T01:35:26.843 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 41s 2022-12-03T01:35:31.845 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:31.930 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:31.931 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 46s 2022-12-03T01:35:31.933 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:32.006 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:32.006 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 47s 2022-12-03T01:35:32.007 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 PodInitializing 0 47s 2022-12-03T01:35:32.007 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m20s 2022-12-03T01:35:32.007 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 47s 2022-12-03T01:35:37.010 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:37.083 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:37.083 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 52s 2022-12-03T01:35:37.085 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:37.158 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:37.158 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 52s 2022-12-03T01:35:37.158 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 PodInitializing 0 52s 2022-12-03T01:35:37.159 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m25s 2022-12-03T01:35:37.159 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 52s 2022-12-03T01:35:42.161 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:42.250 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:42.251 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 57s 2022-12-03T01:35:42.252 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:42.332 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:42.334 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 57s 2022-12-03T01:35:42.334 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 PodInitializing 0 57s 2022-12-03T01:35:42.335 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m30s 2022-12-03T01:35:42.335 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 57s 2022-12-03T01:35:47.338 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:47.418 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:47.418 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 62s 2022-12-03T01:35:47.420 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:47.496 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:47.496 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 62s 2022-12-03T01:35:47.497 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 PodInitializing 0 62s 2022-12-03T01:35:47.497 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m35s 2022-12-03T01:35:47.497 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 62s 2022-12-03T01:35:52.107 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.107138 I | ceph-spec: detected ceph image version: "18.0.0-872 reef" 2022-12-03T01:35:52.108 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.107161 I | ceph-cluster-controller: validating ceph version from provided image 2022-12-03T01:35:52.108 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.107173 W | ceph-cluster-controller: unsupported ceph version detected: "18.0.0-872 reef", pursuing 2022-12-03T01:35:52.130 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.130675 I | ceph-cluster-controller: cluster "rook-ceph": version "18.0.0-872 reef" detected for image "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1" 2022-12-03T01:35:52.212 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.212935 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-12-03 01:35:52.206670052 +0000 UTC m=+76.293777099 LastTransitionTime:2022-12-03 01:35:52.206669889 +0000 UTC m=+76.293776942}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:35:52.256 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.257472 I | op-mon: start running mons 2022-12-03T01:35:52.314 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.314355 I | ceph-spec: creating mon secrets for a new cluster 2022-12-03T01:35:52.334 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.333551 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-12-03T01:35:52.355 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:52.355794 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[],"namespace":""}] data: mapping:{"node":{}} maxMonId:-1 outOfQuorum:] 2022-12-03T01:35:52.499 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:52.581 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:52.581 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 67s 2022-12-03T01:35:52.582 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:52.661 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:52.662 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 PodInitializing 0 67s 2022-12-03T01:35:52.662 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Terminating 0 67s 2022-12-03T01:35:52.663 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m40s 2022-12-03T01:35:52.663 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 67s 2022-12-03T01:35:53.076 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:53.076699 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:35:53.077 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:53.076919 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:35:54.314 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.314255 I | ceph-csi: Detected ceph CSI image version: "v3.7.2" 2022-12-03T01:35:54.324 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323703 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:35:54.324 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323719 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.325 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323723 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:35:54.326 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323726 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.327 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323728 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:35:54.327 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323732 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.327 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323735 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-12-03T01:35:54.328 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323737 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default) 2022-12-03T01:35:54.328 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.323739 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default) 2022-12-03T01:35:54.366 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.367527 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:35:54.367 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.367551 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.367 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.367559 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-12-03T01:35:54.484 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484532 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-12-03T01:35:54.485 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484559 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:35:54.485 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484567 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.485 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484573 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-12-03T01:35:54.486 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484580 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME="" (default) 2022-12-03T01:35:54.486 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.484585 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME_MOUNT="" (default) 2022-12-03T01:35:54.526 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.526097 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:35:54.527 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.526116 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:35:54.527 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.526122 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-12-03T01:35:54.626 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.626582 I | ceph-csi: successfully started CSI CephFS driver 2022-12-03T01:35:54.628 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.626625 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap) 2022-12-03T01:35:54.709 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.708878 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-12-03T01:35:54.710 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.708902 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="File" (configmap) 2022-12-03T01:35:54.852 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.852433 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-12-03T01:35:54.852 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.852456 I | ceph-csi: CSI NFS driver disabled 2022-12-03T01:35:54.853 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.852463 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-12-03T01:35:54.882 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:54.880985 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-12-03T01:35:55.096 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:55.097431 I | op-mon: targeting the mon count 3 2022-12-03T01:35:55.142 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:55.143634 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-12-03T01:35:55.169 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:55.168520 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-12-03T01:35:55.278 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:55.270317 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-12-03T01:35:55.332 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:55.330891 I | ceph-csi: successfully removed CSI NFS driver 2022-12-03T01:35:56.276 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.276669 I | ceph-spec: parsing mon endpoints: 2022-12-03T01:35:56.277 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.276684 W | ceph-spec: ignoring invalid monitor 2022-12-03T01:35:56.277 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.276703 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-12-03T01:35:56.277 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.276708 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-12-03T01:35:56.278 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.276987 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-12-03T01:35:56.278 INFO:tasks.rook.operator.smithi137.stdout:I1203 01:35:56.277072 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-12-03T01:35:56.504 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.505165 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi142 2022-12-03T01:35:56.504 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.505187 I | op-mon: mon a assigned to node smithi142 2022-12-03T01:35:56.677 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.677844 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi155 2022-12-03T01:35:56.678 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.677869 I | op-mon: mon b assigned to node smithi155 2022-12-03T01:35:56.877 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.877746 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi155 2022-12-03T01:35:56.877 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.877772 I | op-mon: mon c assigned to node smithi155 2022-12-03T01:35:56.882 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.883728 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-12-03T01:35:56.963 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:56.964432 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-12-03T01:35:57.005 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.005898 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-12-03T01:35:57.133 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.133021 I | op-mon: creating mon a 2022-12-03T01:35:57.187 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.187448 I | op-mon: mon "a" ip is 10.97.241.124 2022-12-03T01:35:57.667 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:35:57.708 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.707169 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-12-03T01:35:57.709 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.707229 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.241.124:6789"],"namespace":""}] data:a=10.97.241.124:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:-1 outOfQuorum:] 2022-12-03T01:35:57.709 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:57.707282 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-12-03T01:35:57.768 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:57.769 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 72s 2022-12-03T01:35:57.770 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:35:57.923 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:35:57.924 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 0/2 ContainerCreating 0 3s 2022-12-03T01:35:57.924 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 0/5 ContainerCreating 0 3s 2022-12-03T01:35:57.924 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 3s 2022-12-03T01:35:57.925 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 3s 2022-12-03T01:35:57.925 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 0/2 ContainerCreating 0 3s 2022-12-03T01:35:57.926 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 0/2 ContainerCreating 0 3s 2022-12-03T01:35:57.926 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 3s 2022-12-03T01:35:57.927 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 0/5 ContainerCreating 0 3s 2022-12-03T01:35:57.927 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-csi-detect-version-pzzbg 0/1 Terminating 0 72s 2022-12-03T01:35:57.927 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-detect-version-6hpwv 0/1 Terminating 0 72s 2022-12-03T01:35:57.928 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 0/1 Terminating 0 2s 2022-12-03T01:35:57.928 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 0/1 Terminating 0 2s 2022-12-03T01:35:57.928 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 0/1 Terminating 0 2s 2022-12-03T01:35:57.928 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m45s 2022-12-03T01:35:57.929 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 72s 2022-12-03T01:35:58.276 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:58.277164 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:35:58.286 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:58.286406 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:35:58.742 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:58.743413 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-12-03T01:35:58.891 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:58.891247 I | op-mon: updating maxMonID from -1 to 0 2022-12-03T01:35:59.700 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:59.701086 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.241.124:6789"],"namespace":""}] data:a=10.97.241.124:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:0 outOfQuorum:] 2022-12-03T01:35:59.700 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:59.701115 I | op-mon: waiting for mon quorum with [a] 2022-12-03T01:35:59.900 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:59.900275 I | op-mon: mon a is not yet running 2022-12-03T01:35:59.900 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:35:59.900302 I | op-mon: mons running: [] 2022-12-03T01:36:02.930 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:03.003 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:03.004 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 78s 2022-12-03T01:36:03.005 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:03.123 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:03.124 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 0/2 ContainerCreating 0 9s 2022-12-03T01:36:03.124 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 0/5 ContainerCreating 0 9s 2022-12-03T01:36:03.125 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 9s 2022-12-03T01:36:03.125 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 9s 2022-12-03T01:36:03.126 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 9s 2022-12-03T01:36:03.126 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 0/2 ContainerCreating 0 9s 2022-12-03T01:36:03.127 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 9s 2022-12-03T01:36:03.127 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 0/5 ContainerCreating 0 9s 2022-12-03T01:36:03.128 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 5s 2022-12-03T01:36:03.128 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 8s 2022-12-03T01:36:03.128 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 8s 2022-12-03T01:36:03.130 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 8s 2022-12-03T01:36:03.130 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m51s 2022-12-03T01:36:03.131 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 78s 2022-12-03T01:36:08.133 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:08.214 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:08.214 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 83s 2022-12-03T01:36:08.215 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:08.314 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:08.314 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 14s 2022-12-03T01:36:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 0/5 ContainerCreating 0 14s 2022-12-03T01:36:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 14s 2022-12-03T01:36:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 14s 2022-12-03T01:36:08.315 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 14s 2022-12-03T01:36:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 0/2 ContainerCreating 0 14s 2022-12-03T01:36:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 14s 2022-12-03T01:36:08.320 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 0/5 ContainerCreating 0 14s 2022-12-03T01:36:08.321 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 10s 2022-12-03T01:36:08.321 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 13s 2022-12-03T01:36:08.321 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 13s 2022-12-03T01:36:08.321 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 13s 2022-12-03T01:36:08.322 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 2m56s 2022-12-03T01:36:08.322 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 83s 2022-12-03T01:36:13.323 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:13.414 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:13.414 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 88s 2022-12-03T01:36:13.417 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:13.498 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:13.499 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 19s 2022-12-03T01:36:13.499 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 0/5 ContainerCreating 0 19s 2022-12-03T01:36:13.499 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 19s 2022-12-03T01:36:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 19s 2022-12-03T01:36:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 19s 2022-12-03T01:36:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 0/2 ContainerCreating 0 19s 2022-12-03T01:36:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 19s 2022-12-03T01:36:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 0/5 ContainerCreating 0 19s 2022-12-03T01:36:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 15s 2022-12-03T01:36:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 18s 2022-12-03T01:36:13.502 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 18s 2022-12-03T01:36:13.502 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 18s 2022-12-03T01:36:13.502 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m1s 2022-12-03T01:36:13.502 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 88s 2022-12-03T01:36:18.504 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:18.578 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:18.579 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 93s 2022-12-03T01:36:18.580 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:18.661 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:18.662 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 24s 2022-12-03T01:36:18.662 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 0/5 ContainerCreating 0 24s 2022-12-03T01:36:18.662 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 24s 2022-12-03T01:36:18.663 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 24s 2022-12-03T01:36:18.663 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 24s 2022-12-03T01:36:18.664 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 24s 2022-12-03T01:36:18.664 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 24s 2022-12-03T01:36:18.664 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 0/5 ContainerCreating 0 24s 2022-12-03T01:36:18.664 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 20s 2022-12-03T01:36:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 23s 2022-12-03T01:36:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 23s 2022-12-03T01:36:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 23s 2022-12-03T01:36:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m6s 2022-12-03T01:36:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 93s 2022-12-03T01:36:20.026 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:36:20.026287 I | op-mon: mons running: [a] 2022-12-03T01:36:23.669 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:23.747 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:23.748 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 98s 2022-12-03T01:36:23.749 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:23.837 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:23.837 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 29s 2022-12-03T01:36:23.838 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 29s 2022-12-03T01:36:23.838 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 29s 2022-12-03T01:36:23.838 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 29s 2022-12-03T01:36:23.838 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 29s 2022-12-03T01:36:23.839 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 29s 2022-12-03T01:36:23.840 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 29s 2022-12-03T01:36:23.840 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 29s 2022-12-03T01:36:23.840 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 25s 2022-12-03T01:36:23.840 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 28s 2022-12-03T01:36:23.841 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 28s 2022-12-03T01:36:23.841 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 28s 2022-12-03T01:36:23.842 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m11s 2022-12-03T01:36:23.842 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 98s 2022-12-03T01:36:28.845 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:28.946 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:28.947 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 103s 2022-12-03T01:36:28.950 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:29.039 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:29.040 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 35s 2022-12-03T01:36:29.040 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 35s 2022-12-03T01:36:29.041 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 35s 2022-12-03T01:36:29.041 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 35s 2022-12-03T01:36:29.041 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 35s 2022-12-03T01:36:29.042 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 35s 2022-12-03T01:36:29.042 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 35s 2022-12-03T01:36:29.042 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 35s 2022-12-03T01:36:29.043 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 31s 2022-12-03T01:36:29.043 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 34s 2022-12-03T01:36:29.044 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-canary-5db77ddc9b-j2jk8 1/1 Terminating 0 34s 2022-12-03T01:36:29.044 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-canary-7476bc4b54-m49rt 1/1 Terminating 0 34s 2022-12-03T01:36:29.045 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m17s 2022-12-03T01:36:29.045 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 104s 2022-12-03T01:36:34.046 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:34.123 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:34.123 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 109s 2022-12-03T01:36:34.125 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:34.212 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:34.212 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 40s 2022-12-03T01:36:34.213 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 40s 2022-12-03T01:36:34.213 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 40s 2022-12-03T01:36:34.213 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 40s 2022-12-03T01:36:34.213 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 40s 2022-12-03T01:36:34.214 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 40s 2022-12-03T01:36:34.214 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 40s 2022-12-03T01:36:34.214 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 40s 2022-12-03T01:36:34.214 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 36s 2022-12-03T01:36:34.214 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 39s 2022-12-03T01:36:34.215 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m22s 2022-12-03T01:36:34.215 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 109s 2022-12-03T01:36:39.217 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:39.294 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:39.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 114s 2022-12-03T01:36:39.297 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:39.383 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:39.384 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 45s 2022-12-03T01:36:39.385 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 45s 2022-12-03T01:36:39.385 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 45s 2022-12-03T01:36:39.385 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 0/2 ContainerCreating 0 45s 2022-12-03T01:36:39.386 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 45s 2022-12-03T01:36:39.386 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 45s 2022-12-03T01:36:39.387 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 45s 2022-12-03T01:36:39.387 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 45s 2022-12-03T01:36:39.388 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:0/2 0 41s 2022-12-03T01:36:39.388 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-canary-66965ffdc4-czcpq 1/1 Terminating 0 44s 2022-12-03T01:36:39.389 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m27s 2022-12-03T01:36:39.389 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 114s 2022-12-03T01:36:40.168 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:36:40.169660 I | op-mon: mons running: [a] 2022-12-03T01:36:44.392 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:44.467 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:44.468 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 119s 2022-12-03T01:36:44.470 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:44.560 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:44.560 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 50s 2022-12-03T01:36:44.561 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 50s 2022-12-03T01:36:44.561 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 50s 2022-12-03T01:36:44.562 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 50s 2022-12-03T01:36:44.562 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 50s 2022-12-03T01:36:44.562 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 50s 2022-12-03T01:36:44.563 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 50s 2022-12-03T01:36:44.563 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 50s 2022-12-03T01:36:44.563 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Init:1/2 0 46s 2022-12-03T01:36:44.563 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m32s 2022-12-03T01:36:44.564 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 119s 2022-12-03T01:36:49.566 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:49.636 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:49.636 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m4s 2022-12-03T01:36:49.638 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:49.722 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:49.723 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 55s 2022-12-03T01:36:49.724 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 55s 2022-12-03T01:36:49.724 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 0/5 ContainerCreating 0 55s 2022-12-03T01:36:49.726 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 55s 2022-12-03T01:36:49.726 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 55s 2022-12-03T01:36:49.726 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 55s 2022-12-03T01:36:49.726 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 0/5 ContainerCreating 0 55s 2022-12-03T01:36:49.727 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 55s 2022-12-03T01:36:49.727 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Running 0 51s 2022-12-03T01:36:49.727 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m37s 2022-12-03T01:36:49.728 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m4s 2022-12-03T01:36:54.729 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:54.804 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:54.804 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m9s 2022-12-03T01:36:54.806 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:36:54.895 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:54.896 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 60s 2022-12-03T01:36:54.896 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 60s 2022-12-03T01:36:54.896 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 60s 2022-12-03T01:36:54.896 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 60s 2022-12-03T01:36:54.896 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 60s 2022-12-03T01:36:54.897 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 60s 2022-12-03T01:36:54.897 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 60s 2022-12-03T01:36:54.897 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 60s 2022-12-03T01:36:54.897 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 0/1 Running 0 56s 2022-12-03T01:36:54.898 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m42s 2022-12-03T01:36:54.898 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m9s 2022-12-03T01:36:59.900 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:36:59.985 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:36:59.986 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m14s 2022-12-03T01:36:59.988 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:37:00.072 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:37:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 66s 2022-12-03T01:37:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 66s 2022-12-03T01:37:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 66s 2022-12-03T01:37:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 66s 2022-12-03T01:37:00.074 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 66s 2022-12-03T01:37:00.074 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 66s 2022-12-03T01:37:00.074 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 66s 2022-12-03T01:37:00.074 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 66s 2022-12-03T01:37:00.075 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 1/1 Running 0 62s 2022-12-03T01:37:00.075 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m48s 2022-12-03T01:37:00.075 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m15s 2022-12-03T01:37:00.323 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.325344 I | op-mon: mons running: [a] 2022-12-03T01:37:00.548 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.550745 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:37:00.549 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.550769 I | op-mon: mons created: 1 2022-12-03T01:37:00.763 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.765433 I | op-mon: waiting for mon quorum with [a] 2022-12-03T01:37:00.770 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.772774 I | op-mon: mons running: [a] 2022-12-03T01:37:00.994 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.996073 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:37:00.996 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:00.996375 I | op-config: applying ceph settings: 2022-12-03T01:37:00.996 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:00.997 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool delete = true 2022-12-03T01:37:00.997 INFO:tasks.rook.operator.smithi137.stdout:mon cluster log file = 2022-12-03T01:37:00.997 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool size one = true 2022-12-03T01:37:01.263 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.265280 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:01.264 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.265705 I | op-config: applying ceph settings: 2022-12-03T01:37:01.264 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:01.265 INFO:tasks.rook.operator.smithi137.stdout:log to file = false 2022-12-03T01:37:01.549 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.551571 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:01.550 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.551695 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:37:01.818 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.819822 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:37:01.818 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.819841 I | op-mon: creating mon b 2022-12-03T01:37:01.916 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:01.918305 I | op-mon: mon "a" ip is 10.97.241.124 2022-12-03T01:37:02.028 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.030077 I | op-mon: mon "b" ip is 10.101.22.133 2022-12-03T01:37:02.097 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.098082 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.241.124:6789","10.101.22.133:6789"],"namespace":""}] data:a=10.97.241.124:6789,b=10.101.22.133:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:0 outOfQuorum:] 2022-12-03T01:37:02.172 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.174027 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:37:02.172 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.174325 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:37:02.438 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.440417 I | op-mon: 1 of 2 expected mon deployments exist. creating new deployment(s). 2022-12-03T01:37:02.442 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.445366 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-12-03T01:37:02.452 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.454310 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-12-03T01:37:02.663 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:02.663367 I | op-mon: updating maxMonID from 0 to 1 2022-12-03T01:37:03.433 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:03.435869 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.101.22.133:6789","10.97.241.124:6789"],"namespace":""}] data:b=10.101.22.133:6789,a=10.97.241.124:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:1 outOfQuorum:] 2022-12-03T01:37:03.434 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:03.435895 I | op-mon: waiting for mon quorum with [a b] 2022-12-03T01:37:03.826 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:03.828300 I | op-mon: mon b is not yet running 2022-12-03T01:37:03.826 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:03.828327 I | op-mon: mons running: [a] 2022-12-03T01:37:04.053 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:04.055728 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:37:04.054 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:04.055745 I | op-mon: mons created: 2 2022-12-03T01:37:04.319 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:04.321081 I | op-mon: waiting for mon quorum with [a b] 2022-12-03T01:37:04.333 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:04.335236 I | op-mon: mon b is not yet running 2022-12-03T01:37:04.334 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:04.335258 I | op-mon: mons running: [a] 2022-12-03T01:37:05.077 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:37:05.179 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:37:05.180 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m20s 2022-12-03T01:37:05.182 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:37:05.280 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:37:05.280 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 71s 2022-12-03T01:37:05.281 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 71s 2022-12-03T01:37:05.281 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 71s 2022-12-03T01:37:05.281 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 71s 2022-12-03T01:37:05.281 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 71s 2022-12-03T01:37:05.281 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 71s 2022-12-03T01:37:05.282 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 71s 2022-12-03T01:37:05.282 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 71s 2022-12-03T01:37:05.282 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 1/1 Running 0 67s 2022-12-03T01:37:05.283 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-f75f58f99-c9r7b 0/1 Init:1/2 0 3s 2022-12-03T01:37:05.283 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m53s 2022-12-03T01:37:05.283 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 0/1 ContainerCreating 0 2m20s 2022-12-03T01:37:09.354 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:09.350235 I | op-mon: mons running: [a b] 2022-12-03T01:37:10.285 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:37:10.375 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:37:10.375 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Running 0 2m25s 2022-12-03T01:37:10.377 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:37:10.456 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:37:10.457 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 76s 2022-12-03T01:37:10.457 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 76s 2022-12-03T01:37:10.457 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 76s 2022-12-03T01:37:10.457 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 76s 2022-12-03T01:37:10.458 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 76s 2022-12-03T01:37:10.458 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 76s 2022-12-03T01:37:10.458 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 76s 2022-12-03T01:37:10.458 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 76s 2022-12-03T01:37:10.458 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 1/1 Running 0 72s 2022-12-03T01:37:10.459 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-f75f58f99-c9r7b 0/1 Running 0 8s 2022-12-03T01:37:10.459 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 3m58s 2022-12-03T01:37:10.459 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Running 0 2m25s 2022-12-03T01:37:10.460 INFO:tasks.rook:Waiting for 12 OSDs 2022-12-03T01:37:10.461 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:37:13.999 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.001140 I | op-mon: Monitors in quorum: [a b] 2022-12-03T01:37:13.999 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.001715 I | op-config: applying ceph settings: 2022-12-03T01:37:14.000 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:14.000 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool delete = true 2022-12-03T01:37:14.000 INFO:tasks.rook.operator.smithi137.stdout:mon cluster log file = 2022-12-03T01:37:14.000 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool size one = true 2022-12-03T01:37:14.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:37:14.043 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:37:14.044 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:37:14.203 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.205390 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:14.204 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.205658 I | op-config: applying ceph settings: 2022-12-03T01:37:14.204 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:14.205 INFO:tasks.rook.operator.smithi137.stdout:log to file = false 2022-12-03T01:37:14.418 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.420742 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:14.419 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.420832 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:37:14.640 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.642497 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:37:14.640 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.642522 I | op-mon: creating mon c 2022-12-03T01:37:14.733 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.735741 I | op-mon: mon "a" ip is 10.97.241.124 2022-12-03T01:37:14.858 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.860257 I | op-mon: mon "b" ip is 10.101.22.133 2022-12-03T01:37:14.943 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:14.945021 I | op-mon: mon "c" ip is 10.96.71.200 2022-12-03T01:37:15.092 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:15.094120 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.241.124:6789","10.101.22.133:6789","10.96.71.200:6789"],"namespace":""}] data:b=10.101.22.133:6789,c=10.96.71.200:6789,a=10.97.241.124:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:1 outOfQuorum:] 2022-12-03T01:37:15.644 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:15.646157 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:37:15.645 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:15.646409 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:37:16.059 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.061797 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-12-03T01:37:16.065 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.067203 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-12-03T01:37:16.072 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.075457 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-12-03T01:37:16.078 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.080728 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-12-03T01:37:16.086 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.088993 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-12-03T01:37:16.244 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:16.246646 I | op-mon: updating maxMonID from 1 to 2 2022-12-03T01:37:17.069 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.071907 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.241.124:6789","10.101.22.133:6789","10.96.71.200:6789"],"namespace":""}] data:a=10.97.241.124:6789,b=10.101.22.133:6789,c=10.96.71.200:6789 mapping:{"node":{"a":{"Name":"smithi142","Hostname":"smithi142","Address":"172.21.15.142"},"b":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"},"c":{"Name":"smithi155","Hostname":"smithi155","Address":"172.21.15.155"}}} maxMonId:2 outOfQuorum:] 2022-12-03T01:37:17.070 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.071935 I | op-mon: waiting for mon quorum with [a b c] 2022-12-03T01:37:17.648 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.650352 I | op-mon: mon c is not yet running 2022-12-03T01:37:17.648 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.650376 I | op-mon: mons running: [a b] 2022-12-03T01:37:17.878 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.880634 I | op-mon: Monitors in quorum: [a b] 2022-12-03T01:37:17.878 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:17.880650 I | op-mon: mons created: 3 2022-12-03T01:37:18.136 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:18.138654 I | op-mon: waiting for mon quorum with [a b c] 2022-12-03T01:37:18.248 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:18.251282 I | op-mon: mon c is not yet running 2022-12-03T01:37:18.249 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:18.251305 I | op-mon: mons running: [a b] 2022-12-03T01:37:23.269 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:23.270758 I | op-mon: mons running: [a b c] 2022-12-03T01:37:24.046 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:37:27.786 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:37:27.811 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:37:27.811 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:37:27.921 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:27.923475 I | op-mon: Monitors in quorum: [a b c] 2022-12-03T01:37:27.922 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:27.923901 I | op-config: applying ceph settings: 2022-12-03T01:37:27.922 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:27.922 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool delete = true 2022-12-03T01:37:27.922 INFO:tasks.rook.operator.smithi137.stdout:mon cluster log file = 2022-12-03T01:37:27.922 INFO:tasks.rook.operator.smithi137.stdout:mon allow pool size one = true 2022-12-03T01:37:28.144 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.146816 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:28.145 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.147220 I | op-config: applying ceph settings: 2022-12-03T01:37:28.145 INFO:tasks.rook.operator.smithi137.stdout:[global] 2022-12-03T01:37:28.145 INFO:tasks.rook.operator.smithi137.stdout:log to file = false 2022-12-03T01:37:28.357 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.359612 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:37:28.357 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.359702 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:37:28.575 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.577457 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:37:28.575 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.577504 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-12-03T01:37:28.908 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:28.910551 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-12-03T01:37:29.242 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:29.244810 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-12-03T01:37:29.608 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:29.610552 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-12-03T01:37:29.993 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:29.995434 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-12-03T01:37:29.993 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:29.995461 I | cephclient: getting or creating ceph auth key "client.crash" 2022-12-03T01:37:30.376 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.379072 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-12-03T01:37:30.377 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.379103 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-12-03T01:37:30.377 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.379116 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-12-03T01:37:30.699 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.701833 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-12-03T01:37:30.745 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.747715 I | op-mgr: start running mgr 2022-12-03T01:37:30.745 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:30.747749 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-12-03T01:37:31.166 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:31.168786 I | op-config: setting "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-12-03T01:37:31.540 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:31.541513 I | op-config: successfully set "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-12-03T01:37:31.540 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:31.541547 I | op-config: insecure global ID is now disabled 2022-12-03T01:37:31.592 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:31.594716 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:37:37.812 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:37:38.170 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:37:38.187 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:37:38.188 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:37:48.189 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:37:48.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:37:48.752 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:37:48.753 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:37:52.588 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.591033 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-12-03T01:37:52.591 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.594785 I | op-mgr: setting services to point to mgr "a" 2022-12-03T01:37:52.655 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.658339 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-12-03T01:37:52.656 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.658486 I | op-mgr: successful modules: balancer 2022-12-03T01:37:52.746 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.749445 I | op-osd: start running osds in namespace "rook-ceph" 2022-12-03T01:37:52.746 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.749461 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-12-03T01:37:52.764 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.767524 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-12-03T01:37:52.766 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.769887 I | op-osd: creating 12 new PVCs for device set "scratch" 2022-12-03T01:37:52.776 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.779114 I | op-osd: successfully provisioned PVC "scratch-data-0pg2g6" 2022-12-03T01:37:52.813 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.816743 I | op-osd: successfully provisioned PVC "scratch-data-1wnlrq" 2022-12-03T01:37:52.826 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.829134 I | op-osd: successfully provisioned PVC "scratch-data-26t4gh" 2022-12-03T01:37:52.842 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.845721 I | op-osd: successfully provisioned PVC "scratch-data-3gr6bg" 2022-12-03T01:37:52.924 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.921735 I | op-osd: successfully provisioned PVC "scratch-data-4fz98h" 2022-12-03T01:37:52.944 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.945787 I | op-osd: successfully provisioned PVC "scratch-data-5vs6sv" 2022-12-03T01:37:52.975 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:52.978934 I | op-osd: successfully provisioned PVC "scratch-data-6lt9jw" 2022-12-03T01:37:53.015 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.016858 I | op-osd: successfully provisioned PVC "scratch-data-7p8wk8" 2022-12-03T01:37:53.218 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.212736 I | op-osd: successfully provisioned PVC "scratch-data-8gx4vn" 2022-12-03T01:37:53.422 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.425753 I | op-osd: successfully provisioned PVC "scratch-data-9l4jnb" 2022-12-03T01:37:53.542 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.545752 I | op-mgr: successful modules: dashboard 2022-12-03T01:37:53.543 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.545896 I | op-mgr: successful modules: mgr module(s) from the spec 2022-12-03T01:37:53.543 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.546181 I | op-mgr: successful modules: prometheus 2022-12-03T01:37:53.609 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.612386 I | op-osd: successfully provisioned PVC "scratch-data-10fkrdj" 2022-12-03T01:37:53.814 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.817672 I | op-osd: successfully provisioned PVC "scratch-data-11kxmzl" 2022-12-03T01:37:53.818 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:53.821439 I | op-osd: OSD will have its main bluestore block on "scratch-data-0pg2g6" 2022-12-03T01:37:54.232 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:54.235070 I | op-osd: started OSD provisioning job for PVC "scratch-data-0pg2g6" 2022-12-03T01:37:54.232 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:54.235096 I | op-osd: OSD will have its main bluestore block on "scratch-data-1wnlrq" 2022-12-03T01:37:54.641 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:54.644077 I | op-osd: started OSD provisioning job for PVC "scratch-data-1wnlrq" 2022-12-03T01:37:54.641 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:54.644115 I | op-osd: OSD will have its main bluestore block on "scratch-data-26t4gh" 2022-12-03T01:37:55.076 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.077195 I | op-osd: started OSD provisioning job for PVC "scratch-data-26t4gh" 2022-12-03T01:37:55.077 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.077220 I | op-osd: OSD will have its main bluestore block on "scratch-data-3gr6bg" 2022-12-03T01:37:55.533 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.536939 I | op-osd: started OSD provisioning job for PVC "scratch-data-3gr6bg" 2022-12-03T01:37:55.534 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.536960 I | op-osd: OSD will have its main bluestore block on "scratch-data-4fz98h" 2022-12-03T01:37:55.851 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.854695 I | op-osd: started OSD provisioning job for PVC "scratch-data-4fz98h" 2022-12-03T01:37:55.852 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:55.854723 I | op-osd: OSD will have its main bluestore block on "scratch-data-5vs6sv" 2022-12-03T01:37:56.319 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:56.321773 I | op-osd: started OSD provisioning job for PVC "scratch-data-5vs6sv" 2022-12-03T01:37:56.319 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:56.321810 I | op-osd: OSD will have its main bluestore block on "scratch-data-6lt9jw" 2022-12-03T01:37:56.644 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:56.647629 I | op-osd: started OSD provisioning job for PVC "scratch-data-6lt9jw" 2022-12-03T01:37:56.645 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:56.647657 I | op-osd: OSD will have its main bluestore block on "scratch-data-7p8wk8" 2022-12-03T01:37:57.037 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.040528 I | op-osd: started OSD provisioning job for PVC "scratch-data-7p8wk8" 2022-12-03T01:37:57.037 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.040562 I | op-osd: OSD will have its main bluestore block on "scratch-data-8gx4vn" 2022-12-03T01:37:57.514 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.516952 I | op-osd: started OSD provisioning job for PVC "scratch-data-8gx4vn" 2022-12-03T01:37:57.514 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.516987 I | op-osd: OSD will have its main bluestore block on "scratch-data-9l4jnb" 2022-12-03T01:37:57.903 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.905800 I | op-osd: started OSD provisioning job for PVC "scratch-data-9l4jnb" 2022-12-03T01:37:57.903 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:57.905828 I | op-osd: OSD will have its main bluestore block on "scratch-data-10fkrdj" 2022-12-03T01:37:58.352 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.351481 I | op-osd: started OSD provisioning job for PVC "scratch-data-10fkrdj" 2022-12-03T01:37:58.353 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.351500 I | op-osd: OSD will have its main bluestore block on "scratch-data-11kxmzl" 2022-12-03T01:37:58.659 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.660798 I | op-osd: started OSD provisioning job for PVC "scratch-data-11kxmzl" 2022-12-03T01:37:58.660 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.660857 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-12-03T01:37:58.660 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.660866 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-12-03T01:37:58.754 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:37:58.799 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800530 I | op-osd: OSD orchestration status for PVC scratch-data-0pg2g6 is "starting" 2022-12-03T01:37:58.799 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800567 I | op-osd: OSD orchestration status for PVC scratch-data-10fkrdj is "starting" 2022-12-03T01:37:58.800 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800590 I | op-osd: OSD orchestration status for PVC scratch-data-11kxmzl is "starting" 2022-12-03T01:37:58.800 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800607 I | op-osd: OSD orchestration status for PVC scratch-data-1wnlrq is "starting" 2022-12-03T01:37:58.801 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800618 I | op-osd: OSD orchestration status for PVC scratch-data-26t4gh is "starting" 2022-12-03T01:37:58.801 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800634 I | op-osd: OSD orchestration status for PVC scratch-data-3gr6bg is "starting" 2022-12-03T01:37:58.801 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800653 I | op-osd: OSD orchestration status for PVC scratch-data-4fz98h is "starting" 2022-12-03T01:37:58.802 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800677 I | op-osd: OSD orchestration status for PVC scratch-data-5vs6sv is "starting" 2022-12-03T01:37:58.802 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800692 I | op-osd: OSD orchestration status for PVC scratch-data-6lt9jw is "starting" 2022-12-03T01:37:58.803 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800706 I | op-osd: OSD orchestration status for PVC scratch-data-7p8wk8 is "starting" 2022-12-03T01:37:58.803 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800721 I | op-osd: OSD orchestration status for PVC scratch-data-8gx4vn is "starting" 2022-12-03T01:37:58.804 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:37:58.800741 I | op-osd: OSD orchestration status for PVC scratch-data-9l4jnb is "starting" 2022-12-03T01:37:59.265 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:37:59.295 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:37:59.296 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:38:04.527 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:04.530156 I | op-mgr: successful modules: orchestrator modules 2022-12-03T01:38:08.429 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:08.432973 I | op-osd: OSD orchestration status for node scratch-data-3gr6bg is "orchestrating" 2022-12-03T01:38:08.796 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:08.799761 I | op-osd: OSD orchestration status for PVC scratch-data-3gr6bg is "orchestrating" 2022-12-03T01:38:09.297 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:38:10.348 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:38:10.377 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:38:10.377 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:38:10.896 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:10.899388 I | op-osd: OSD orchestration status for node scratch-data-4fz98h is "orchestrating" 2022-12-03T01:38:11.014 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.017397 I | op-osd: OSD orchestration status for node scratch-data-6lt9jw is "orchestrating" 2022-12-03T01:38:11.029 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.033258 I | op-osd: OSD orchestration status for node scratch-data-0pg2g6 is "orchestrating" 2022-12-03T01:38:11.421 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.425134 I | op-osd: OSD orchestration status for PVC scratch-data-0pg2g6 is "orchestrating" 2022-12-03T01:38:11.438 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.441792 I | op-osd: OSD orchestration status for PVC scratch-data-6lt9jw is "orchestrating" 2022-12-03T01:38:11.446 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.449957 I | op-osd: OSD orchestration status for PVC scratch-data-4fz98h is "orchestrating" 2022-12-03T01:38:11.963 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:11.966683 I | op-osd: OSD orchestration status for node scratch-data-1wnlrq is "orchestrating" 2022-12-03T01:38:12.388 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:12.391799 I | op-osd: OSD orchestration status for node scratch-data-7p8wk8 is "orchestrating" 2022-12-03T01:38:12.463 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:12.466726 I | op-osd: OSD orchestration status for PVC scratch-data-1wnlrq is "orchestrating" 2022-12-03T01:38:12.805 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:12.808297 I | op-osd: OSD orchestration status for PVC scratch-data-7p8wk8 is "orchestrating" 2022-12-03T01:38:13.447 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:13.450232 I | op-osd: OSD orchestration status for node scratch-data-26t4gh is "orchestrating" 2022-12-03T01:38:13.714 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:13.717043 I | op-osd: OSD orchestration status for node scratch-data-5vs6sv is "orchestrating" 2022-12-03T01:38:13.846 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:13.850209 I | op-osd: OSD orchestration status for PVC scratch-data-26t4gh is "orchestrating" 2022-12-03T01:38:14.105 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:14.108929 I | op-osd: OSD orchestration status for PVC scratch-data-5vs6sv is "orchestrating" 2022-12-03T01:38:16.809 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:16.811971 I | op-osd: OSD orchestration status for PVC scratch-data-3gr6bg is "completed" 2022-12-03T01:38:16.810 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:16.811998 I | op-osd: creating OSD 0 on PVC "scratch-data-3gr6bg" 2022-12-03T01:38:16.810 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:16.812011 I | op-osd: OSD will have its main bluestore block on "scratch-data-3gr6bg" 2022-12-03T01:38:17.423 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:17.426418 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:38:17.472 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:17.475639 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi155": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:38:19.025 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.027499 I | op-osd: OSD orchestration status for PVC scratch-data-0pg2g6 is "completed" 2022-12-03T01:38:19.026 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.027527 I | op-osd: creating OSD 1 on PVC "scratch-data-0pg2g6" 2022-12-03T01:38:19.026 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.027543 I | op-osd: OSD will have its main bluestore block on "scratch-data-0pg2g6" 2022-12-03T01:38:19.239 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.242884 I | op-osd: OSD orchestration status for PVC scratch-data-1wnlrq is "completed" 2022-12-03T01:38:19.240 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.242902 I | op-osd: creating OSD 3 on PVC "scratch-data-1wnlrq" 2022-12-03T01:38:19.241 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.242909 I | op-osd: OSD will have its main bluestore block on "scratch-data-1wnlrq" 2022-12-03T01:38:19.737 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.741070 I | op-osd: OSD orchestration status for PVC scratch-data-4fz98h is "completed" 2022-12-03T01:38:19.738 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.741090 I | op-osd: creating OSD 2 on PVC "scratch-data-4fz98h" 2022-12-03T01:38:19.738 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:19.741098 I | op-osd: OSD will have its main bluestore block on "scratch-data-4fz98h" 2022-12-03T01:38:20.118 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.070599 I | op-osd: OSD orchestration status for PVC scratch-data-6lt9jw is "completed" 2022-12-03T01:38:20.118 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.070626 I | op-osd: creating OSD 4 on PVC "scratch-data-6lt9jw" 2022-12-03T01:38:20.118 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.070638 I | op-osd: OSD will have its main bluestore block on "scratch-data-6lt9jw" 2022-12-03T01:38:20.379 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:38:20.735 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.738315 I | op-osd: OSD orchestration status for PVC scratch-data-26t4gh is "completed" 2022-12-03T01:38:20.735 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.738358 I | op-osd: creating OSD 6 on PVC "scratch-data-26t4gh" 2022-12-03T01:38:20.736 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:20.738378 I | op-osd: OSD will have its main bluestore block on "scratch-data-26t4gh" 2022-12-03T01:38:21.087 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.089888 I | op-osd: OSD orchestration status for PVC scratch-data-7p8wk8 is "completed" 2022-12-03T01:38:21.087 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.089906 I | op-osd: creating OSD 5 on PVC "scratch-data-7p8wk8" 2022-12-03T01:38:21.087 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.089913 I | op-osd: OSD will have its main bluestore block on "scratch-data-7p8wk8" 2022-12-03T01:38:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:38:21.299 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:38:21.300 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:38:21.581 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.580116 I | op-osd: OSD orchestration status for PVC scratch-data-5vs6sv is "completed" 2022-12-03T01:38:21.582 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.580158 I | op-osd: creating OSD 7 on PVC "scratch-data-5vs6sv" 2022-12-03T01:38:21.582 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:21.580180 I | op-osd: OSD will have its main bluestore block on "scratch-data-5vs6sv" 2022-12-03T01:38:31.301 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:38:32.832 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:38:32.857 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:38:32.858 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:38:42.861 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:38:43.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:38:43.771 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:38:43.772 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:38:53.774 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:38:54.343 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:38:54.363 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:38:54.364 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:58.664 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:38:58.660942 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:39:04.366 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:04.964 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:04.964 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:14.965 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:15.488 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:15.508 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:15.508 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:25.509 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:26.018 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:26.037 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:26.038 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:36.039 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:36.574 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:36.595 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:36.596 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:46.598 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:47.151 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:47.168 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:47.173 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:57.174 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:39:57.758 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:39:57.781 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:39:57.782 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:58.660 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:39:58.661734 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:40:07.783 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:40:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:40:08.288 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:40:08.289 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:18.290 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:40:18.853 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:40:18.876 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:40:18.877 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:28.879 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:40:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:40:29.421 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:40:29.422 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:39.423 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:40:40.068 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:40:40.090 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:40:40.091 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:50.092 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:40:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:40:50.627 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:40:50.628 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:58.664 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:40:58.661276 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:41:00.630 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:01.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:01.395 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:01.395 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:11.396 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:11.972 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:11.992 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:11.993 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:21.995 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:22.559 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:22.559 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:32.561 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:33.088 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:33.114 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:33.115 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:43.117 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:43.619 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:43.642 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:43.642 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:53.644 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:41:54.176 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:41:54.199 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:41:54.201 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:58.664 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:41:58.660934 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:42:04.207 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:04.752 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:04.781 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:04.781 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:14.783 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:15.401 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:15.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:15.429 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:25.430 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:25.925 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:25.948 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:25.948 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:35.950 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:36.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:36.487 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:36.488 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:46.490 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:47.057 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:47.058 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:57.059 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:42:57.563 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:42:57.619 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:42:57.620 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:58.663 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:42:58.661226 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:43:07.624 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:43:08.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:43:08.137 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:43:08.137 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:18.138 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:43:18.648 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:43:18.669 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:43:18.669 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:28.671 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:43:29.193 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:43:29.214 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:43:29.215 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:39.218 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:43:39.762 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:43:39.790 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:43:39.790 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:49.792 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:43:50.395 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:43:50.433 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:43:50.434 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:58.662 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:43:58.661966 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:44:00.436 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:00.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:00.970 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:00.971 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:10.973 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:11.488 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:11.489 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:21.491 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:21.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:22.024 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:22.025 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:32.027 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:32.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:32.630 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:32.630 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:42.631 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:43.158 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:43.184 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:43.185 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:53.188 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:44:53.699 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:44:53.720 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:44:53.721 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:44:58.660947 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:45:03.722 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:04.220 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:04.244 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:04.245 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:14.247 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:14.773 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:14.793 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:14.793 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:24.794 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:25.448 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:25.449 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:35.450 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:35.966 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:35.989 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:35.990 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:45.991 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:46.527 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:46.528 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:56.529 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:45:57.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:45:57.155 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:45:57.156 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:45:58.661038 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:46:07.157 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:46:07.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:46:07.686 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:46:07.687 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:17.689 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:46:18.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:46:18.232 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:46:18.233 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:28.235 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:46:28.765 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:46:28.790 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:46:28.791 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:38.793 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:46:39.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:46:39.350 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:46:39.352 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:49.354 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:46:49.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:46:49.881 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:46:49.882 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:58.669 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:46:58.661219 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:46:59.884 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:00.389 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:00.413 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:00.413 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:10.414 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:10.964 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:10.987 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:10.988 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:20.990 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:21.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:21.550 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:21.550 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:31.552 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:32.040 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:32.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:32.064 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:42.066 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:42.562 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:42.584 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:42.585 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:52.586 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:47:53.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:47:53.133 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:47:53.134 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:47:58.662 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:47:58.661835 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:48:03.135 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:03.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:03.665 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:03.665 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:13.667 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:14.182 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:14.207 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:14.207 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:24.208 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:24.727 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:24.745 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:24.745 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:34.746 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:35.424 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:35.447 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:35.448 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:45.449 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:45.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:45.973 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:45.973 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:55.975 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:48:56.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:48:56.521 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:48:56.521 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:48:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:48:58.661322 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:49:06.523 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:07.053 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:07.053 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:49:17.055 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:17.554 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:17.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:17.575 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:49:27.577 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:28.185 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:28.209 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:28.210 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:49:38.211 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:38.721 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:38.744 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:38.745 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:49:48.747 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:49.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:49.281 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:49.282 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:49:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:49:58.661057 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:49:59.283 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:49:59.892 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:49:59.913 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:49:59.913 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:09.915 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:50:10.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:50:10.446 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:50:10.447 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:20.448 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:50:20.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:50:20.990 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:50:20.991 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:30.993 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:50:31.517 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:50:31.539 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:50:31.540 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:41.541 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:50:42.044 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:50:42.069 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:50:42.069 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:52.071 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:50:52.649 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:50:52.673 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:50:52.674 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:50:58.664 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:50:58.660976 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:51:02.677 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:03.183 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:03.204 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:03.205 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:13.206 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:13.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:13.725 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:13.726 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:23.727 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:24.214 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:24.228 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:24.229 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:34.230 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:34.740 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:34.762 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:34.762 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:44.763 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:45.323 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:45.323 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:55.325 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:51:55.838 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:51:55.865 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:51:55.866 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:51:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:51:58.661529 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:52:05.867 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:06.446 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:06.447 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:16.448 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:16.985 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:17.009 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:17.009 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:27.010 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:27.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:27.807 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:27.807 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:37.808 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:38.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:38.348 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:38.348 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:48.350 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:48.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:48.877 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:58.661 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:58.661027 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:52:58.878 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-fwpqn -- ceph osd stat -f json 2022-12-03T01:52:59.414 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:52:59.434 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":18,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031531,"num_in_osds":8,"osd_in_since":1670031495,"num_remapped_pgs":0} 2022-12-03T01:52:59.435 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:52:59.436 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:52:59.437 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 471, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:52:59.450 DEBUG:teuthology.orchestra.remote:smithi137:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:52:59.504 DEBUG:teuthology.orchestra.run.smithi137:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-12-03T01:52:59.599 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps "rook-ceph-tools" deleted 2022-12-03T01:52:59.630 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 403, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:52:59.631 DEBUG:teuthology.orchestra.run.smithi137:> kubectl delete -f cluster.yaml 2022-12-03T01:52:59.722 INFO:teuthology.orchestra.run.smithi137.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-12-03T01:52:59.723 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.722736 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-12-03T01:52:59.723 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.722840 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-12-03T01:52:59.724 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.722909 I | op-osd: context cancelled, exiting OSD update and create loop 2022-12-03T01:52:59.724 INFO:tasks.rook.operator.smithi137.stdout:I1203 01:52:59.723347 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"="context canceled" 2022-12-03T01:52:59.725 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.726088 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-12-03T01:52:59.726 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.726112 I | operator: watching all namespaces for Ceph CRs 2022-12-03T01:52:59.726 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.726142 I | operator: setting up schemes 2022-12-03T01:52:59.727 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.727778 I | operator: setting up the controller-runtime manager 2022-12-03T01:52:59.737 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.735015 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-12-03T01:52:59.787 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787873 I | operator: failed to get cert manager 2022-12-03T01:52:59.788 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787905 I | ceph-cluster-controller: successfully started 2022-12-03T01:52:59.790 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787946 I | ceph-cluster-controller: enabling hotplug orchestration 2022-12-03T01:52:59.790 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787957 I | ceph-crashcollector-controller: successfully started 2022-12-03T01:52:59.790 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787971 I | ceph-block-pool-controller: successfully started 2022-12-03T01:52:59.790 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787986 I | ceph-object-store-user-controller: successfully started 2022-12-03T01:52:59.791 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.787999 I | ceph-object-realm-controller: successfully started 2022-12-03T01:52:59.791 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788008 I | ceph-object-zonegroup-controller: successfully started 2022-12-03T01:52:59.791 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788016 I | ceph-object-zone-controller: successfully started 2022-12-03T01:52:59.791 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788136 I | ceph-object-controller: successfully started 2022-12-03T01:52:59.792 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788178 I | ceph-file-controller: successfully started 2022-12-03T01:52:59.792 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788212 I | ceph-nfs-controller: successfully started 2022-12-03T01:52:59.792 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788246 I | ceph-rbd-mirror-controller: successfully started 2022-12-03T01:52:59.792 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788275 I | ceph-client-controller: successfully started 2022-12-03T01:52:59.792 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788302 I | ceph-filesystem-mirror-controller: successfully started 2022-12-03T01:52:59.793 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788340 I | operator: rook-ceph-operator-config-controller successfully started 2022-12-03T01:52:59.793 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788367 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-12-03T01:52:59.793 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788387 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-12-03T01:52:59.793 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788408 I | ceph-bucket-topic: successfully started 2022-12-03T01:52:59.794 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788419 I | ceph-bucket-notification: successfully started 2022-12-03T01:52:59.794 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788430 I | ceph-bucket-notification: successfully started 2022-12-03T01:52:59.794 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788449 I | ceph-fs-subvolumegroup-controller: successfully started 2022-12-03T01:52:59.794 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.788460 I | blockpool-rados-namespace-controller: successfully started 2022-12-03T01:52:59.794 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.789765 I | operator: starting the controller-runtime manager 2022-12-03T01:52:59.795 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.791617 I | ceph-cluster-controller: context cancelled, exiting reconcile 2022-12-03T01:52:59.795 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.792181 I | operator: successfully started the controller-runtime manager 2022-12-03T01:52:59.895 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.895151 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-12-03T01:52:59.895 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.895224 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-12-03T01:52:59.896 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.895246 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-12-03T01:52:59.896 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.896655 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:52:59.929 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.929255 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap) 2022-12-03T01:52:59.929 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.929275 I | operator: rook-ceph-operator-config-controller done reconciling 2022-12-03T01:52:59.969 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:52:59.969923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" can be deleted safely. deleting CephCluster "rook-ceph/rook-ceph" 2022-12-03T01:53:00.005 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.006013 I | ceph-cluster-controller: cleaning up CephCluster "rook-ceph/rook-ceph" 2022-12-03T01:53:00.006 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.006037 I | ceph-cluster-controller: checking any PVC created by drivers "rook-ceph.cephfs.csi.ceph.com" and "rook-ceph.rbd.csi.ceph.com" with clusterID "rook-ceph" 2022-12-03T01:53:00.011 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010663 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi137-lv-1" 2022-12-03T01:53:00.011 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010677 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi137-lv-2" 2022-12-03T01:53:00.011 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010683 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi137-lv-3" 2022-12-03T01:53:00.012 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010689 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi137-lv-4" 2022-12-03T01:53:00.012 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010694 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi142-lv-1" 2022-12-03T01:53:00.012 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010700 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi142-lv-2" 2022-12-03T01:53:00.012 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010706 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi142-lv-3" 2022-12-03T01:53:00.013 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010713 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi142-lv-4" 2022-12-03T01:53:00.013 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010718 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi155-lv-1" 2022-12-03T01:53:00.013 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010724 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi155-lv-2" 2022-12-03T01:53:00.013 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010730 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi155-lv-3" 2022-12-03T01:53:00.014 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010736 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi155-lv-4" 2022-12-03T01:53:00.014 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010746 I | ceph-cluster-controller: no volume attachments for cluster "rook-ceph" 2022-12-03T01:53:00.014 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.010777 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon" 2022-12-03T01:53:00.023 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.024215 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon-endpoints" 2022-12-03T01:53:00.076 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.077050 I | ceph-spec: removing finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-12-03T01:53:00.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.101162 E | clusterdisruption-controller: cephcluster "rook-ceph/rook-ceph" seems to be deleted, not requeuing until triggered again 2022-12-03T01:53:00.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.102146 I | ceph-csi: CSI Ceph RBD driver disabled 2022-12-03T01:53:00.102 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.102161 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-12-03T01:53:00.106 INFO:tasks.rook:Waiting for cluster to stop 2022-12-03T01:53:00.106 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:53:00.154 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.154869 I | op-k8sutil: Removed daemonset csi-rbdplugin 2022-12-03T01:53:00.239 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.238381 I | op-k8sutil: "csi-rbdplugin" still found. waiting... 2022-12-03T01:53:00.249 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.249988 I | ceph-spec: object "rook-ceph-osd-3" matched on delete, reconciling 2022-12-03T01:53:00.270 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.270902 I | ceph-spec: object "rook-csi-cephfs-provisioner" matched on delete, reconciling 2022-12-03T01:53:00.271 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.271827 I | ceph-spec: object "rook-csi-rbd-node" matched on delete, reconciling 2022-12-03T01:53:00.272 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.271892 I | ceph-spec: object "rook-ceph-mon" matched on delete, reconciling 2022-12-03T01:53:00.273 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.273418 I | ceph-spec: object "rook-ceph-osd-4" matched on delete, reconciling 2022-12-03T01:53:00.273 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.273791 I | ceph-spec: object "rook-ceph-osd-5" matched on delete, reconciling 2022-12-03T01:53:00.273 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.274016 I | ceph-spec: object "rook-ceph-crashcollector-smithi155" matched on delete, reconciling 2022-12-03T01:53:00.274 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.274311 I | ceph-spec: object "rook-ceph-osd-2" matched on delete, reconciling 2022-12-03T01:53:00.288 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:53:00.289 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 17m 2022-12-03T01:53:00.289 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 17m 2022-12-03T01:53:00.289 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 17m 2022-12-03T01:53:00.289 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 17m 2022-12-03T01:53:00.290 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Running 0 17m 2022-12-03T01:53:00.290 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-fbgn7 2/2 Running 0 17m 2022-12-03T01:53:00.290 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 17m 2022-12-03T01:53:00.290 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 17m 2022-12-03T01:53:00.291 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-crashcollector-smithi142-dd468dc9c-nnwxs 1/1 Running 0 15m 2022-12-03T01:53:00.291 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-crashcollector-smithi155-75d8c68d97-g9z4b 1/1 Running 0 14m 2022-12-03T01:53:00.291 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mgr-a-ff667f8f-wv987 1/1 Running 0 15m 2022-12-03T01:53:00.291 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 1/1 Running 0 17m 2022-12-03T01:53:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-f75f58f99-c9r7b 1/1 Running 0 15m 2022-12-03T01:53:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-f7f9d5959-fkkn2 1/1 Running 0 15m 2022-12-03T01:53:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 19m 2022-12-03T01:53:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-0-8564b8857f-9pd2x 1/1 Running 0 14m 2022-12-03T01:53:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-1-745f856b6b-w9mn5 1/1 Running 0 14m 2022-12-03T01:53:00.293 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-2-84b7d69f68-bmrbk 1/1 Running 0 14m 2022-12-03T01:53:00.293 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-3-79578b87d7-sfhpj 1/1 Running 0 14m 2022-12-03T01:53:00.293 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-4-6cf9b944d8-96g6w 1/1 Running 0 14m 2022-12-03T01:53:00.293 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-5-7cb997556d-qhwk6 1/1 Running 0 14m 2022-12-03T01:53:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-6-847f467fdc-6b59j 1/1 Running 0 14m 2022-12-03T01:53:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-7-555cd5c57b-g2cbj 1/1 Running 0 14m 2022-12-03T01:53:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-0pg2g6-n5r8r 0/1 Completed 0 15m 2022-12-03T01:53:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-10fkrdj-hqwh6 0/1 Pending 0 15m 2022-12-03T01:53:00.294 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-11kxmzl-qqks2 0/1 Pending 0 15m 2022-12-03T01:53:00.295 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-1wnlrq-qmz89 0/1 Completed 0 15m 2022-12-03T01:53:00.295 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-26t4gh-k9ztw 0/1 Completed 0 15m 2022-12-03T01:53:00.295 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-3gr6bg-98pwm 0/1 Completed 0 15m 2022-12-03T01:53:00.296 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-4fz98h-bzgl2 0/1 Completed 0 15m 2022-12-03T01:53:00.296 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-5vs6sv-xmkgt 0/1 Completed 0 15m 2022-12-03T01:53:00.296 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-6lt9jw-5vvn8 0/1 Completed 0 15m 2022-12-03T01:53:00.296 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-7p8wk8-78f88 0/1 Completed 0 15m 2022-12-03T01:53:00.296 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-8gx4vn-r7tvf 0/1 Pending 0 15m 2022-12-03T01:53:00.297 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-prepare-scratch-data-9l4jnb-rj9pc 0/1 Pending 0 15m 2022-12-03T01:53:00.297 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Terminating 0 18m 2022-12-03T01:53:00.858 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.858321 I | ceph-spec: object "cluster-peer-token-rook-ceph" matched on delete, reconciling 2022-12-03T01:53:00.933 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:00.934135 I | ceph-spec: object "rook-ceph-mgr-a-keyring" matched on delete, reconciling 2022-12-03T01:53:01.100 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.100956 I | ceph-spec: object "rook-ceph-config" matched on delete, reconciling 2022-12-03T01:53:01.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.102425 I | ceph-spec: object "rook-ceph-osd-7" matched on delete, reconciling 2022-12-03T01:53:01.210 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.210224 I | ceph-spec: object "rook-ceph-mgr" matched on delete, reconciling 2022-12-03T01:53:01.217 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.217878 I | ceph-spec: object "rook-ceph-crashcollector-smithi142" matched on delete, reconciling 2022-12-03T01:53:01.292 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.293105 I | ceph-spec: object "rook-ceph-mon-endpoints" matched on delete, reconciling 2022-12-03T01:53:01.368 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.368288 I | ceph-spec: object "rook-csi-cephfs-node" matched on delete, reconciling 2022-12-03T01:53:01.956 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.956507 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-12-03T01:53:01.968 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.968747 I | ceph-spec: object "rook-ceph-mon-c" matched on delete, reconciling 2022-12-03T01:53:01.975 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.976435 I | ceph-spec: object "rook-csi-rbd-provisioner" matched on delete, reconciling 2022-12-03T01:53:01.993 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:01.994083 I | ceph-spec: object "rook-ceph-mgr-a" matched on delete, reconciling 2022-12-03T01:53:02.101 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.101725 I | ceph-spec: object "rook-ceph-mons-keyring" matched on delete, reconciling 2022-12-03T01:53:02.219 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.219346 I | ceph-spec: object "rook-ceph-mon-a" matched on delete, reconciling 2022-12-03T01:53:02.236 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.236560 I | ceph-spec: object "rook-ceph-osd-6" matched on delete, reconciling 2022-12-03T01:53:02.273 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.273945 I | ceph-spec: object "rook-ceph-osd-1" matched on delete, reconciling 2022-12-03T01:53:02.293 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.293816 I | ceph-spec: object "rook-ceph-admin-keyring" matched on delete, reconciling 2022-12-03T01:53:02.310 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.311264 I | ceph-spec: object "rook-ceph-mon-b" matched on delete, reconciling 2022-12-03T01:53:02.351 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:02.352125 I | ceph-spec: object "rook-ceph-crash-collector-keyring" matched on delete, reconciling 2022-12-03T01:53:05.298 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:53:05.417 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:53:05.417 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Running 0 17m 2022-12-03T01:53:05.417 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 17m 2022-12-03T01:53:05.418 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 17m 2022-12-03T01:53:05.418 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Running 0 17m 2022-12-03T01:53:05.418 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-26nlc 2/2 Terminating 0 17m 2022-12-03T01:53:05.419 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-fcx26 5/5 Running 0 17m 2022-12-03T01:53:05.419 INFO:teuthology.orchestra.run.smithi137.stdout:csi-rbdplugin-provisioner-56d69f5d8-tmpp5 5/5 Running 0 17m 2022-12-03T01:53:05.420 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-crashcollector-smithi142-dd468dc9c-nnwxs 1/1 Terminating 0 15m 2022-12-03T01:53:05.420 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-crashcollector-smithi155-75d8c68d97-g9z4b 1/1 Terminating 0 14m 2022-12-03T01:53:05.421 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mgr-a-ff667f8f-wv987 1/1 Terminating 0 15m 2022-12-03T01:53:05.421 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-a-68cf89b79-xrfwx 1/1 Terminating 0 17m 2022-12-03T01:53:05.421 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-b-f75f58f99-c9r7b 1/1 Terminating 0 16m 2022-12-03T01:53:05.421 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-mon-c-f7f9d5959-fkkn2 1/1 Terminating 0 15m 2022-12-03T01:53:05.421 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 19m 2022-12-03T01:53:05.422 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-0-8564b8857f-9pd2x 1/1 Terminating 0 14m 2022-12-03T01:53:05.422 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-1-745f856b6b-w9mn5 1/1 Terminating 0 14m 2022-12-03T01:53:05.422 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-2-84b7d69f68-bmrbk 1/1 Terminating 0 14m 2022-12-03T01:53:05.422 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-3-79578b87d7-sfhpj 1/1 Terminating 0 14m 2022-12-03T01:53:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-4-6cf9b944d8-96g6w 1/1 Terminating 0 14m 2022-12-03T01:53:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-5-7cb997556d-qhwk6 1/1 Terminating 0 14m 2022-12-03T01:53:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-6-847f467fdc-6b59j 1/1 Terminating 0 14m 2022-12-03T01:53:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-7-555cd5c57b-g2cbj 1/1 Terminating 0 14m 2022-12-03T01:53:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Terminating 0 18m 2022-12-03T01:53:06.282 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:06.282522 I | op-k8sutil: confirmed csi-rbdplugin does not exist 2022-12-03T01:53:06.283 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:06.282548 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-12-03T01:53:06.316 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:06.316773 I | op-k8sutil: Removed deployment csi-rbdplugin-provisioner 2022-12-03T01:53:06.319 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:06.319925 I | op-k8sutil: "csi-rbdplugin-provisioner" still found. waiting... 2022-12-03T01:53:10.328 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.328980 I | op-k8sutil: confirmed csi-rbdplugin-provisioner does not exist 2022-12-03T01:53:10.347 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.347884 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-12-03T01:53:10.348 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.347913 I | ceph-csi: CSI CephFS driver disabled 2022-12-03T01:53:10.348 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.347925 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-12-03T01:53:10.370 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.370212 I | op-k8sutil: Removed daemonset csi-cephfsplugin 2022-12-03T01:53:10.372 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:10.373276 I | op-k8sutil: "csi-cephfsplugin" still found. waiting... 2022-12-03T01:53:10.424 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:53:10.520 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:53:10.521 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-k4454 2/2 Terminating 0 17m 2022-12-03T01:53:10.521 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Running 0 17m 2022-12-03T01:53:10.522 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Running 0 17m 2022-12-03T01:53:10.522 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-w8jtg 2/2 Terminating 0 17m 2022-12-03T01:53:10.522 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 19m 2022-12-03T01:53:10.523 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-0-8564b8857f-9pd2x 1/1 Terminating 0 14m 2022-12-03T01:53:10.524 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-1-745f856b6b-w9mn5 1/1 Terminating 0 14m 2022-12-03T01:53:10.524 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-2-84b7d69f68-bmrbk 1/1 Terminating 0 14m 2022-12-03T01:53:10.524 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-3-79578b87d7-sfhpj 1/1 Terminating 0 14m 2022-12-03T01:53:10.525 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-4-6cf9b944d8-96g6w 1/1 Terminating 0 14m 2022-12-03T01:53:10.525 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-5-7cb997556d-qhwk6 1/1 Terminating 0 14m 2022-12-03T01:53:10.526 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-osd-7-555cd5c57b-g2cbj 1/1 Terminating 0 14m 2022-12-03T01:53:10.526 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Terminating 0 18m 2022-12-03T01:53:14.381 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:14.380885 I | op-k8sutil: confirmed csi-cephfsplugin does not exist 2022-12-03T01:53:14.382 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:14.380914 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-12-03T01:53:14.415 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:14.415162 I | op-k8sutil: Removed deployment csi-cephfsplugin-provisioner 2022-12-03T01:53:14.419 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:14.420047 I | op-k8sutil: "csi-cephfsplugin-provisioner" still found. waiting... 2022-12-03T01:53:15.529 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph get pods 2022-12-03T01:53:15.605 INFO:teuthology.orchestra.run.smithi137.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:53:15.605 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-9zj98 5/5 Terminating 0 17m 2022-12-03T01:53:15.606 INFO:teuthology.orchestra.run.smithi137.stdout:csi-cephfsplugin-provisioner-75875b5887-h2vqr 5/5 Terminating 0 17m 2022-12-03T01:53:15.606 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-operator-5ddcc89457-2gd84 1/1 Running 0 20m 2022-12-03T01:53:15.606 INFO:teuthology.orchestra.run.smithi137.stdout:rook-ceph-tools-5679b7d8f-fwpqn 1/1 Terminating 0 18m 2022-12-03T01:53:15.608 DEBUG:teuthology.orchestra.run.smithi137:> kubectl -n rook-ceph delete configmap rook-config-override 2022-12-03T01:53:15.732 INFO:teuthology.orchestra.run.smithi137.stdout:configmap "rook-config-override" deleted 2022-12-03T01:53:15.744 DEBUG:teuthology.orchestra.run.smithi137:> rm -f cluster.yaml 2022-12-03T01:53:15.750 INFO:tasks.rook:Checking cluster log for badness... 2022-12-03T01:53:15.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/lib/rook/rook-ceph/log/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2022-12-03T01:53:15.811 INFO:teuthology.orchestra.run.smithi137.stderr:grep: /var/lib/rook/rook-ceph/log/ceph.log: No such file or directory 2022-12-03T01:53:15.812 INFO:tasks.rook:Compressing logs... 2022-12-03T01:53:15.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:15.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:15.859 DEBUG:teuthology.orchestra.run.smithi155:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:15.866 INFO:teuthology.orchestra.run.smithi137.stderr:find: '/var/lib/rook/rook-ceph/log': No such file or directory 2022-12-03T01:53:16.874 INFO:tasks.rook:Archiving logs... 2022-12-03T01:53:16.877 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/ubuntu@smithi137.front.sepia.ceph.com/log 2022-12-03T01:53:16.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:53:16.893 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/rook/rook-ceph/log: Cannot open: No such file or directory 2022-12-03T01:53:16.894 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2022-12-03T01:53:16.896 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/ubuntu@smithi142.front.sepia.ceph.com/log 2022-12-03T01:53:16.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:53:17.145 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/ubuntu@smithi155.front.sepia.ceph.com/log 2022-12-03T01:53:17.146 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:53:17.402 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:53:17.403 INFO:tasks.rook:Cleaning up rook operator 2022-12-03T01:53:17.404 DEBUG:teuthology.orchestra.run.smithi137:> kubectl delete -f operator.yaml 2022-12-03T01:53:17.491 INFO:teuthology.orchestra.run.smithi137.stdout:configmap "rook-ceph-operator-config" deleted 2022-12-03T01:53:17.499 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps "rook-ceph-operator" deleted 2022-12-03T01:53:17.506 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf rook operator.yaml 2022-12-03T01:53:17.581 INFO:tasks.rook.operator.smithi137.stdout:2022-12-03 01:53:17.581040 I | operator: shutdown signal received, exiting... context canceled 2022-12-03T01:53:18.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf /var/lib/rook 2022-12-03T01:53:18.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf /var/lib/rook 2022-12-03T01:53:18.671 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf /var/lib/rook 2022-12-03T01:53:18.693 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:53:18.831 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=338664706450452e848acdede7af5d19 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:53:18.834 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-12-03T01:53:18.847 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2022-12-03T01:53:18.897 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi137:/dev/vg_nvme/lv_1... 2022-12-03T01:53:18.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo nvme disconnect -n lv_1 2022-12-03T01:53:19.190 INFO:teuthology.orchestra.run.smithi137.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:53:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:19.192 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi137:/dev/vg_nvme/lv_2... 2022-12-03T01:53:19.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo nvme disconnect -n lv_2 2022-12-03T01:53:19.506 INFO:teuthology.orchestra.run.smithi137.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:53:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:19.508 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi137:/dev/vg_nvme/lv_3... 2022-12-03T01:53:19.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo nvme disconnect -n lv_3 2022-12-03T01:53:19.846 INFO:teuthology.orchestra.run.smithi137.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:53:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:19.850 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi137:/dev/vg_nvme/lv_4... 2022-12-03T01:53:19.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo nvme disconnect -n lv_4 2022-12-03T01:53:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:20.152 INFO:teuthology.orchestra.run.smithi137.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:53:20.153 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-12-03T01:53:20.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/scratch_devs 2022-12-03T01:53:20.170 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi142:/dev/vg_nvme/lv_1... 2022-12-03T01:53:20.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme disconnect -n lv_1 2022-12-03T01:53:20.465 INFO:teuthology.orchestra.run.smithi142.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:53:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:20.468 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi142:/dev/vg_nvme/lv_2... 2022-12-03T01:53:20.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme disconnect -n lv_2 2022-12-03T01:53:20.781 INFO:teuthology.orchestra.run.smithi142.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:53:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:20.784 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi142:/dev/vg_nvme/lv_3... 2022-12-03T01:53:20.784 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme disconnect -n lv_3 2022-12-03T01:53:21.077 INFO:teuthology.orchestra.run.smithi142.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:53:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:21.080 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi142:/dev/vg_nvme/lv_4... 2022-12-03T01:53:21.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo nvme disconnect -n lv_4 2022-12-03T01:53:21.413 INFO:teuthology.orchestra.run.smithi142.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:53:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:21.415 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2022-12-03T01:53:21.416 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/scratch_devs 2022-12-03T01:53:21.430 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi155:/dev/vg_nvme/lv_1... 2022-12-03T01:53:21.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme disconnect -n lv_1 2022-12-03T01:53:21.707 INFO:teuthology.orchestra.run.smithi155.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:53:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:21.709 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi155:/dev/vg_nvme/lv_2... 2022-12-03T01:53:21.710 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme disconnect -n lv_2 2022-12-03T01:53:22.007 INFO:teuthology.orchestra.run.smithi155.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:53:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:22.009 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi155:/dev/vg_nvme/lv_3... 2022-12-03T01:53:22.010 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme disconnect -n lv_3 2022-12-03T01:53:22.311 INFO:teuthology.orchestra.run.smithi155.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:53:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:22.314 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi155:/dev/vg_nvme/lv_4... 2022-12-03T01:53:22.315 DEBUG:teuthology.orchestra.run.smithi155:> sudo nvme disconnect -n lv_4 2022-12-03T01:53:22.615 INFO:teuthology.orchestra.run.smithi155.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:53:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:22.617 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2022-12-03T01:53:22.618 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/scratch_devs 2022-12-03T01:53:22.632 DEBUG:teuthology.run_tasks:Unwinding manager kubeadm 2022-12-03T01:53:22.647 INFO:tasks.kubeadm:Tearing down kubeadm 2022-12-03T01:53:22.648 DEBUG:teuthology.orchestra.run.smithi137:> kubectl delete installation default 2022-12-03T01:53:22.771 INFO:teuthology.orchestra.run.smithi137.stdout:installation.operator.tigera.io "default" deleted 2022-12-03T01:53:26.428 DEBUG:teuthology.orchestra.run.smithi137:> kubectl delete -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 2022-12-03T01:53:27.026 INFO:teuthology.orchestra.run.smithi137.stdout:namespace "tigera-operator" deleted 2022-12-03T01:53:27.328 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" deleted 2022-12-03T01:53:27.511 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "bgppeers.crd.projectcalico.org" deleted 2022-12-03T01:53:27.647 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org" deleted 2022-12-03T01:53:27.865 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "caliconodestatuses.crd.projectcalico.org" deleted 2022-12-03T01:53:28.097 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" deleted 2022-12-03T01:53:28.243 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" deleted 2022-12-03T01:53:28.382 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" deleted 2022-12-03T01:53:28.469 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org" deleted 2022-12-03T01:53:28.572 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org" deleted 2022-12-03T01:53:28.680 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" deleted 2022-12-03T01:53:28.707 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org" deleted 2022-12-03T01:53:28.822 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org" deleted 2022-12-03T01:53:28.882 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" deleted 2022-12-03T01:53:28.983 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "ipreservations.crd.projectcalico.org" deleted 2022-12-03T01:53:29.051 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "kubecontrollersconfigurations.crd.projectcalico.org" deleted 2022-12-03T01:53:29.165 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" deleted 2022-12-03T01:53:29.274 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "networksets.crd.projectcalico.org" deleted 2022-12-03T01:53:29.341 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "apiservers.operator.tigera.io" deleted 2022-12-03T01:53:29.366 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "imagesets.operator.tigera.io" deleted 2022-12-03T01:53:29.551 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "installations.operator.tigera.io" deleted 2022-12-03T01:53:29.618 INFO:teuthology.orchestra.run.smithi137.stdout:customresourcedefinition.apiextensions.k8s.io "tigerastatuses.operator.tigera.io" deleted 2022-12-03T01:53:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:serviceaccount "tigera-operator" deleted 2022-12-03T01:53:29.685 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrole.rbac.authorization.k8s.io "tigera-operator" deleted 2022-12-03T01:53:29.747 INFO:teuthology.orchestra.run.smithi137.stdout:clusterrolebinding.rbac.authorization.k8s.io "tigera-operator" deleted 2022-12-03T01:53:29.763 INFO:teuthology.orchestra.run.smithi137.stdout:deployment.apps "tigera-operator" deleted 2022-12-03T01:53:35.964 INFO:tasks.kubeadm:Deconfiguring kubectl 2022-12-03T01:53:35.965 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:53:35.987 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:53:36.005 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:53:36.024 INFO:tasks.kubeadm:Cleaning up node 2022-12-03T01:53:36.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:53:36.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:53:36.050 DEBUG:teuthology.orchestra.run.smithi155:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:53:36.081 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Reading configuration from the cluster... 2022-12-03T01:53:36.082 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:53:36.090 INFO:teuthology.orchestra.run.smithi142.stdout:[preflight] Running pre-flight checks 2022-12-03T01:53:36.091 INFO:teuthology.orchestra.run.smithi142.stderr:W1203 01:53:36.091339 48130 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-12-03T01:53:36.092 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] No etcd config found. Assuming external etcd 2022-12-03T01:53:36.092 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-12-03T01:53:36.093 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Stopping the kubelet service 2022-12-03T01:53:36.109 INFO:teuthology.orchestra.run.smithi155.stdout:[preflight] Running pre-flight checks 2022-12-03T01:53:36.109 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] No etcd config found. Assuming external etcd 2022-12-03T01:53:36.110 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-12-03T01:53:36.110 INFO:teuthology.orchestra.run.smithi155.stderr:W1203 01:53:36.108575 56873 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-12-03T01:53:36.111 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Stopping the kubelet service 2022-12-03T01:53:36.114 INFO:teuthology.orchestra.run.smithi137.stdout:[preflight] Running pre-flight checks 2022-12-03T01:53:36.141 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Stopping the kubelet service 2022-12-03T01:53:36.184 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:53:36.201 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:53:36.206 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:53:42.868 INFO:teuthology.orchestra.run.smithi137.stderr:W1203 01:53:42.864823 32590 cleanupnode.go:94] [reset] Failed to remove containers: [failed to stop running pod 9d692943fdf7ab31daf72dad4609c5389b0d055af1699b7a676edd6621e9196a: output: E1203 01:53:36.591620 32754 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"9d692943fdf7ab31daf72dad4609c5389b0d055af1699b7a676edd6621e9196a\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="9d692943fdf7ab31daf72dad4609c5389b0d055af1699b7a676edd6621e9196a" 2022-12-03T01:53:42.868 INFO:teuthology.orchestra.run.smithi137.stderr:time="2022-12-03T01:53:36Z" level=fatal msg="stopping the pod sandbox \"9d692943fdf7ab31daf72dad4609c5389b0d055af1699b7a676edd6621e9196a\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"9d692943fdf7ab31daf72dad4609c5389b0d055af1699b7a676edd6621e9196a\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:42.869 INFO:teuthology.orchestra.run.smithi137.stderr:: exit status 1, failed to stop running pod 45470d436624c215cddde56257d449005ff4ba79512492f532b3d448c5980a77: output: E1203 01:53:37.882975 33003 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"45470d436624c215cddde56257d449005ff4ba79512492f532b3d448c5980a77\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="45470d436624c215cddde56257d449005ff4ba79512492f532b3d448c5980a77" 2022-12-03T01:53:42.869 INFO:teuthology.orchestra.run.smithi137.stderr:time="2022-12-03T01:53:37Z" level=fatal msg="stopping the pod sandbox \"45470d436624c215cddde56257d449005ff4ba79512492f532b3d448c5980a77\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"45470d436624c215cddde56257d449005ff4ba79512492f532b3d448c5980a77\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:42.869 INFO:teuthology.orchestra.run.smithi137.stderr:: exit status 1, failed to stop running pod 4490cdacc908d720ea4b0dee2a7e164793d07db4e4903d3281bada3dc6f69cd3: output: E1203 01:53:38.901311 33277 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"4490cdacc908d720ea4b0dee2a7e164793d07db4e4903d3281bada3dc6f69cd3\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="4490cdacc908d720ea4b0dee2a7e164793d07db4e4903d3281bada3dc6f69cd3" 2022-12-03T01:53:42.870 INFO:teuthology.orchestra.run.smithi137.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"4490cdacc908d720ea4b0dee2a7e164793d07db4e4903d3281bada3dc6f69cd3\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"4490cdacc908d720ea4b0dee2a7e164793d07db4e4903d3281bada3dc6f69cd3\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:42.870 INFO:teuthology.orchestra.run.smithi137.stderr:: exit status 1] 2022-12-03T01:53:42.871 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:53:42.871 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:53:42.872 INFO:teuthology.orchestra.run.smithi137.stdout:[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet] 2022-12-03T01:53:42.882 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:53:42.883 INFO:teuthology.orchestra.run.smithi137.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:53:42.884 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:53:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:53:42.885 INFO:teuthology.orchestra.run.smithi137.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:53:42.885 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:53:42.886 INFO:teuthology.orchestra.run.smithi137.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:53:42.886 INFO:teuthology.orchestra.run.smithi137.stdout:to reset your system's IPVS tables. 2022-12-03T01:53:42.887 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-12-03T01:53:42.888 INFO:teuthology.orchestra.run.smithi137.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:53:42.888 INFO:teuthology.orchestra.run.smithi137.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:53:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:W1203 01:53:43.814602 48130 cleanupnode.go:94] [reset] Failed to remove containers: [failed to stop running pod 2d2e0bf7344c82fde2d2f6301ab8e8127c80848b57a06bd6caa52d90441338d6: output: E1203 01:53:36.560409 48296 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"2d2e0bf7344c82fde2d2f6301ab8e8127c80848b57a06bd6caa52d90441338d6\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="2d2e0bf7344c82fde2d2f6301ab8e8127c80848b57a06bd6caa52d90441338d6" 2022-12-03T01:53:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:36Z" level=fatal msg="stopping the pod sandbox \"2d2e0bf7344c82fde2d2f6301ab8e8127c80848b57a06bd6caa52d90441338d6\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"2d2e0bf7344c82fde2d2f6301ab8e8127c80848b57a06bd6caa52d90441338d6\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 66cf9e84a724342c4c8ca443e10712eebdb6bd1d9f8e63e6b5a71a774f597b02: output: E1203 01:53:36.917353 48475 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"66cf9e84a724342c4c8ca443e10712eebdb6bd1d9f8e63e6b5a71a774f597b02\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="66cf9e84a724342c4c8ca443e10712eebdb6bd1d9f8e63e6b5a71a774f597b02" 2022-12-03T01:53:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:36Z" level=fatal msg="stopping the pod sandbox \"66cf9e84a724342c4c8ca443e10712eebdb6bd1d9f8e63e6b5a71a774f597b02\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"66cf9e84a724342c4c8ca443e10712eebdb6bd1d9f8e63e6b5a71a774f597b02\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 12d78a64f720376a3935b5d10a968d7457fd66acd2492974743d568df0312271: output: E1203 01:53:37.339235 48654 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"12d78a64f720376a3935b5d10a968d7457fd66acd2492974743d568df0312271\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="12d78a64f720376a3935b5d10a968d7457fd66acd2492974743d568df0312271" 2022-12-03T01:53:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:37Z" level=fatal msg="stopping the pod sandbox \"12d78a64f720376a3935b5d10a968d7457fd66acd2492974743d568df0312271\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"12d78a64f720376a3935b5d10a968d7457fd66acd2492974743d568df0312271\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 49137f35b2f1924657c9fb2ad3d9a38b30e90202e830fd0d23439cf34b3d824b: output: E1203 01:53:37.854059 48835 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"49137f35b2f1924657c9fb2ad3d9a38b30e90202e830fd0d23439cf34b3d824b\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="49137f35b2f1924657c9fb2ad3d9a38b30e90202e830fd0d23439cf34b3d824b" 2022-12-03T01:53:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:37Z" level=fatal msg="stopping the pod sandbox \"49137f35b2f1924657c9fb2ad3d9a38b30e90202e830fd0d23439cf34b3d824b\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"49137f35b2f1924657c9fb2ad3d9a38b30e90202e830fd0d23439cf34b3d824b\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 9bc12a57efda1f34487c64225146a1701a41ae546947c3e09ae52ffbdb1d4b62: output: E1203 01:53:38.216830 49013 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"9bc12a57efda1f34487c64225146a1701a41ae546947c3e09ae52ffbdb1d4b62\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="9bc12a57efda1f34487c64225146a1701a41ae546947c3e09ae52ffbdb1d4b62" 2022-12-03T01:53:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"9bc12a57efda1f34487c64225146a1701a41ae546947c3e09ae52ffbdb1d4b62\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"9bc12a57efda1f34487c64225146a1701a41ae546947c3e09ae52ffbdb1d4b62\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 9de9748b5ee54c7c74dc0cdfcc5e7e64415833327ab283f19e62cdfc69b71cdf: output: E1203 01:53:38.579737 49192 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"9de9748b5ee54c7c74dc0cdfcc5e7e64415833327ab283f19e62cdfc69b71cdf\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="9de9748b5ee54c7c74dc0cdfcc5e7e64415833327ab283f19e62cdfc69b71cdf" 2022-12-03T01:53:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"9de9748b5ee54c7c74dc0cdfcc5e7e64415833327ab283f19e62cdfc69b71cdf\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"9de9748b5ee54c7c74dc0cdfcc5e7e64415833327ab283f19e62cdfc69b71cdf\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod b83405808c0c16455288b2e38edcc4f8299e3546cbeb8b4275e7afc8c2ac0203: output: E1203 01:53:38.955310 49371 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"b83405808c0c16455288b2e38edcc4f8299e3546cbeb8b4275e7afc8c2ac0203\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="b83405808c0c16455288b2e38edcc4f8299e3546cbeb8b4275e7afc8c2ac0203" 2022-12-03T01:53:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"b83405808c0c16455288b2e38edcc4f8299e3546cbeb8b4275e7afc8c2ac0203\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"b83405808c0c16455288b2e38edcc4f8299e3546cbeb8b4275e7afc8c2ac0203\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod d692b4f2b7621e8d375e9828142be62b40081f02f96be71fa08488bc3a1a3198: output: E1203 01:53:39.304339 49554 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"d692b4f2b7621e8d375e9828142be62b40081f02f96be71fa08488bc3a1a3198\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="d692b4f2b7621e8d375e9828142be62b40081f02f96be71fa08488bc3a1a3198" 2022-12-03T01:53:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:39Z" level=fatal msg="stopping the pod sandbox \"d692b4f2b7621e8d375e9828142be62b40081f02f96be71fa08488bc3a1a3198\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"d692b4f2b7621e8d375e9828142be62b40081f02f96be71fa08488bc3a1a3198\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod bbf3bae8a76d52db2fb4d970ec55ad6d18a7c1d187e3251ec206f4a47778527c: output: E1203 01:53:39.894957 49733 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"bbf3bae8a76d52db2fb4d970ec55ad6d18a7c1d187e3251ec206f4a47778527c\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="bbf3bae8a76d52db2fb4d970ec55ad6d18a7c1d187e3251ec206f4a47778527c" 2022-12-03T01:53:43.821 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:39Z" level=fatal msg="stopping the pod sandbox \"bbf3bae8a76d52db2fb4d970ec55ad6d18a7c1d187e3251ec206f4a47778527c\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"bbf3bae8a76d52db2fb4d970ec55ad6d18a7c1d187e3251ec206f4a47778527c\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.821 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 6061a94177c5f518322342f5cfc5e717897f81f740d39a15785f7ecb0bdee892: output: E1203 01:53:40.265814 49913 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"6061a94177c5f518322342f5cfc5e717897f81f740d39a15785f7ecb0bdee892\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="6061a94177c5f518322342f5cfc5e717897f81f740d39a15785f7ecb0bdee892" 2022-12-03T01:53:43.822 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:40Z" level=fatal msg="stopping the pod sandbox \"6061a94177c5f518322342f5cfc5e717897f81f740d39a15785f7ecb0bdee892\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"6061a94177c5f518322342f5cfc5e717897f81f740d39a15785f7ecb0bdee892\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.822 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 6285dc6b4cc571f37374943a850893f36596eba16916a8bb67a430030a30a4c2: output: E1203 01:53:40.617674 50094 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"6285dc6b4cc571f37374943a850893f36596eba16916a8bb67a430030a30a4c2\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="6285dc6b4cc571f37374943a850893f36596eba16916a8bb67a430030a30a4c2" 2022-12-03T01:53:43.822 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:40Z" level=fatal msg="stopping the pod sandbox \"6285dc6b4cc571f37374943a850893f36596eba16916a8bb67a430030a30a4c2\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"6285dc6b4cc571f37374943a850893f36596eba16916a8bb67a430030a30a4c2\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:43.823 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 200f5cb233271dc2b7fb09d3efca69b204a8199d79bccdd5e5ed030ed9576ecc: output: E1203 01:53:41.969170 50272 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"200f5cb233271dc2b7fb09d3efca69b204a8199d79bccdd5e5ed030ed9576ecc\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="200f5cb233271dc2b7fb09d3efca69b204a8199d79bccdd5e5ed030ed9576ecc" 2022-12-03T01:53:43.823 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:41Z" level=fatal msg="stopping the pod sandbox \"200f5cb233271dc2b7fb09d3efca69b204a8199d79bccdd5e5ed030ed9576ecc\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"200f5cb233271dc2b7fb09d3efca69b204a8199d79bccdd5e5ed030ed9576ecc\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:43.824 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1, failed to stop running pod 5e64f6a6ca798696fa874596b1b370d2c722a99b89f7503d4966b16a95ecf1b8: output: E1203 01:53:42.495906 50489 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"5e64f6a6ca798696fa874596b1b370d2c722a99b89f7503d4966b16a95ecf1b8\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="5e64f6a6ca798696fa874596b1b370d2c722a99b89f7503d4966b16a95ecf1b8" 2022-12-03T01:53:43.824 INFO:teuthology.orchestra.run.smithi142.stderr:time="2022-12-03T01:53:42Z" level=fatal msg="stopping the pod sandbox \"5e64f6a6ca798696fa874596b1b370d2c722a99b89f7503d4966b16a95ecf1b8\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"5e64f6a6ca798696fa874596b1b370d2c722a99b89f7503d4966b16a95ecf1b8\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:43.825 INFO:teuthology.orchestra.run.smithi142.stderr:: exit status 1] 2022-12-03T01:53:43.827 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:53:43.827 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:53:43.827 INFO:teuthology.orchestra.run.smithi142.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet] 2022-12-03T01:53:43.828 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:53:43.828 INFO:teuthology.orchestra.run.smithi142.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:53:43.828 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:53:43.829 INFO:teuthology.orchestra.run.smithi142.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:53:43.829 INFO:teuthology.orchestra.run.smithi142.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:53:43.829 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:53:43.830 INFO:teuthology.orchestra.run.smithi142.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:53:43.830 INFO:teuthology.orchestra.run.smithi142.stdout:to reset your system's IPVS tables. 2022-12-03T01:53:43.830 INFO:teuthology.orchestra.run.smithi142.stdout: 2022-12-03T01:53:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:53:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:53:44.438 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:53:44.439 INFO:teuthology.orchestra.run.smithi155.stderr:W1203 01:53:44.437584 56873 cleanupnode.go:94] [reset] Failed to remove containers: [failed to stop running pod 713c000bf935f8e7106db6d0144fadcf7f471cf58200535f6d5abfdcdac2da40: output: E1203 01:53:36.608076 57037 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"713c000bf935f8e7106db6d0144fadcf7f471cf58200535f6d5abfdcdac2da40\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="713c000bf935f8e7106db6d0144fadcf7f471cf58200535f6d5abfdcdac2da40" 2022-12-03T01:53:44.440 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:36Z" level=fatal msg="stopping the pod sandbox \"713c000bf935f8e7106db6d0144fadcf7f471cf58200535f6d5abfdcdac2da40\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"713c000bf935f8e7106db6d0144fadcf7f471cf58200535f6d5abfdcdac2da40\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.440 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 4b8d718ad5ede2457e28220a240f5af85695686a264aaf96e2075032cdd4e9cc: output: E1203 01:53:36.973089 57217 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"4b8d718ad5ede2457e28220a240f5af85695686a264aaf96e2075032cdd4e9cc\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="4b8d718ad5ede2457e28220a240f5af85695686a264aaf96e2075032cdd4e9cc" 2022-12-03T01:53:44.441 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:36Z" level=fatal msg="stopping the pod sandbox \"4b8d718ad5ede2457e28220a240f5af85695686a264aaf96e2075032cdd4e9cc\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"4b8d718ad5ede2457e28220a240f5af85695686a264aaf96e2075032cdd4e9cc\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.442 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 8804af093a30dddda55a50792ac876edd4a3f4723b0d0ecb5249a820096dc1b6: output: E1203 01:53:37.380917 57396 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"8804af093a30dddda55a50792ac876edd4a3f4723b0d0ecb5249a820096dc1b6\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="8804af093a30dddda55a50792ac876edd4a3f4723b0d0ecb5249a820096dc1b6" 2022-12-03T01:53:44.442 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:37Z" level=fatal msg="stopping the pod sandbox \"8804af093a30dddda55a50792ac876edd4a3f4723b0d0ecb5249a820096dc1b6\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"8804af093a30dddda55a50792ac876edd4a3f4723b0d0ecb5249a820096dc1b6\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.443 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 30f2e8f0736e02c246bf4c79e830e3a602df04882b90e20fabf440e0ea9f7d36: output: E1203 01:53:37.850857 57574 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"30f2e8f0736e02c246bf4c79e830e3a602df04882b90e20fabf440e0ea9f7d36\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="30f2e8f0736e02c246bf4c79e830e3a602df04882b90e20fabf440e0ea9f7d36" 2022-12-03T01:53:44.443 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:37Z" level=fatal msg="stopping the pod sandbox \"30f2e8f0736e02c246bf4c79e830e3a602df04882b90e20fabf440e0ea9f7d36\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"30f2e8f0736e02c246bf4c79e830e3a602df04882b90e20fabf440e0ea9f7d36\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.444 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 2eb3143ff842eb6dab3f8d4a9336e142600a4e213bee8109f27dc15bc926c5ed: output: E1203 01:53:38.214057 57756 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"2eb3143ff842eb6dab3f8d4a9336e142600a4e213bee8109f27dc15bc926c5ed\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="2eb3143ff842eb6dab3f8d4a9336e142600a4e213bee8109f27dc15bc926c5ed" 2022-12-03T01:53:44.444 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"2eb3143ff842eb6dab3f8d4a9336e142600a4e213bee8109f27dc15bc926c5ed\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"2eb3143ff842eb6dab3f8d4a9336e142600a4e213bee8109f27dc15bc926c5ed\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.444 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 1029b90138e8acfe9cf66f18b9f6f21d5ea488c2ee998d114b780d3cb86673cb: output: E1203 01:53:38.578191 57937 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"1029b90138e8acfe9cf66f18b9f6f21d5ea488c2ee998d114b780d3cb86673cb\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="1029b90138e8acfe9cf66f18b9f6f21d5ea488c2ee998d114b780d3cb86673cb" 2022-12-03T01:53:44.445 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"1029b90138e8acfe9cf66f18b9f6f21d5ea488c2ee998d114b780d3cb86673cb\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"1029b90138e8acfe9cf66f18b9f6f21d5ea488c2ee998d114b780d3cb86673cb\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.445 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 7427b876547666d1a7418d80f414876825b74df5b89aa43a6e1598e6e9774cd4: output: E1203 01:53:38.949133 58117 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"7427b876547666d1a7418d80f414876825b74df5b89aa43a6e1598e6e9774cd4\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="7427b876547666d1a7418d80f414876825b74df5b89aa43a6e1598e6e9774cd4" 2022-12-03T01:53:44.446 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:38Z" level=fatal msg="stopping the pod sandbox \"7427b876547666d1a7418d80f414876825b74df5b89aa43a6e1598e6e9774cd4\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"7427b876547666d1a7418d80f414876825b74df5b89aa43a6e1598e6e9774cd4\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.446 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 2701c714ee35edb39c28772a175189b3122e974398b1b345988f496db6503b1e: output: E1203 01:53:39.296342 58297 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"2701c714ee35edb39c28772a175189b3122e974398b1b345988f496db6503b1e\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="2701c714ee35edb39c28772a175189b3122e974398b1b345988f496db6503b1e" 2022-12-03T01:53:44.446 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:39Z" level=fatal msg="stopping the pod sandbox \"2701c714ee35edb39c28772a175189b3122e974398b1b345988f496db6503b1e\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"2701c714ee35edb39c28772a175189b3122e974398b1b345988f496db6503b1e\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod b877f34abc969f71151b006f0cad10a29e11bc6d255eb6ac8be6da248e9472f2: output: E1203 01:53:39.896706 58476 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"b877f34abc969f71151b006f0cad10a29e11bc6d255eb6ac8be6da248e9472f2\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="b877f34abc969f71151b006f0cad10a29e11bc6d255eb6ac8be6da248e9472f2" 2022-12-03T01:53:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:39Z" level=fatal msg="stopping the pod sandbox \"b877f34abc969f71151b006f0cad10a29e11bc6d255eb6ac8be6da248e9472f2\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"b877f34abc969f71151b006f0cad10a29e11bc6d255eb6ac8be6da248e9472f2\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.447 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 663566a8c2f7bce1607539c2ed94f5ce20a33ee999e16692e853a0afb4be42ba: output: E1203 01:53:40.262674 58656 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"663566a8c2f7bce1607539c2ed94f5ce20a33ee999e16692e853a0afb4be42ba\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="663566a8c2f7bce1607539c2ed94f5ce20a33ee999e16692e853a0afb4be42ba" 2022-12-03T01:53:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:40Z" level=fatal msg="stopping the pod sandbox \"663566a8c2f7bce1607539c2ed94f5ce20a33ee999e16692e853a0afb4be42ba\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"663566a8c2f7bce1607539c2ed94f5ce20a33ee999e16692e853a0afb4be42ba\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.448 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod f793775b40ab0e002b04abf067f328fc2bd7ff63dd375ee00fe6f110e40026a5: output: E1203 01:53:40.622252 58840 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"f793775b40ab0e002b04abf067f328fc2bd7ff63dd375ee00fe6f110e40026a5\": error getting ClusterInformation: connection is unauthorized: Unauthorized" podSandboxID="f793775b40ab0e002b04abf067f328fc2bd7ff63dd375ee00fe6f110e40026a5" 2022-12-03T01:53:44.449 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:40Z" level=fatal msg="stopping the pod sandbox \"f793775b40ab0e002b04abf067f328fc2bd7ff63dd375ee00fe6f110e40026a5\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"f793775b40ab0e002b04abf067f328fc2bd7ff63dd375ee00fe6f110e40026a5\": error getting ClusterInformation: connection is unauthorized: Unauthorized" 2022-12-03T01:53:44.449 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 42c1d43702ee4dfa48c644bea70d796618693de5a603e5cb771c461e92d24201: output: E1203 01:53:41.978844 59023 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"42c1d43702ee4dfa48c644bea70d796618693de5a603e5cb771c461e92d24201\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="42c1d43702ee4dfa48c644bea70d796618693de5a603e5cb771c461e92d24201" 2022-12-03T01:53:44.449 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:41Z" level=fatal msg="stopping the pod sandbox \"42c1d43702ee4dfa48c644bea70d796618693de5a603e5cb771c461e92d24201\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"42c1d43702ee4dfa48c644bea70d796618693de5a603e5cb771c461e92d24201\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:44.450 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 899604b2f6d3a15c33b007d950e2a0f974b58445aa5af83b449fdf7133c564bf: output: E1203 01:53:42.283872 59201 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"899604b2f6d3a15c33b007d950e2a0f974b58445aa5af83b449fdf7133c564bf\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="899604b2f6d3a15c33b007d950e2a0f974b58445aa5af83b449fdf7133c564bf" 2022-12-03T01:53:44.451 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:42Z" level=fatal msg="stopping the pod sandbox \"899604b2f6d3a15c33b007d950e2a0f974b58445aa5af83b449fdf7133c564bf\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"899604b2f6d3a15c33b007d950e2a0f974b58445aa5af83b449fdf7133c564bf\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:44.451 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod f8a21de28ffaf74087de1523969d5776e637879cdf04b39b4f3a8f48cd0df20d: output: E1203 01:53:42.604568 59379 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"f8a21de28ffaf74087de1523969d5776e637879cdf04b39b4f3a8f48cd0df20d\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="f8a21de28ffaf74087de1523969d5776e637879cdf04b39b4f3a8f48cd0df20d" 2022-12-03T01:53:44.452 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:42Z" level=fatal msg="stopping the pod sandbox \"f8a21de28ffaf74087de1523969d5776e637879cdf04b39b4f3a8f48cd0df20d\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"f8a21de28ffaf74087de1523969d5776e637879cdf04b39b4f3a8f48cd0df20d\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:44.452 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1, failed to stop running pod 21f90de1088a69deb47a8993929b27a9d751cb10abd0e49f383a84605b27d243: output: E1203 01:53:42.913612 59560 remote_runtime.go:269] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"21f90de1088a69deb47a8993929b27a9d751cb10abd0e49f383a84605b27d243\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" podSandboxID="21f90de1088a69deb47a8993929b27a9d751cb10abd0e49f383a84605b27d243" 2022-12-03T01:53:44.453 INFO:teuthology.orchestra.run.smithi155.stderr:time="2022-12-03T01:53:42Z" level=fatal msg="stopping the pod sandbox \"21f90de1088a69deb47a8993929b27a9d751cb10abd0e49f383a84605b27d243\": rpc error: code = Unknown desc = failed to destroy network for sandbox \"21f90de1088a69deb47a8993929b27a9d751cb10abd0e49f383a84605b27d243\": error getting ClusterInformation: Get \"https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default\": dial tcp 10.96.0.1:443: connect: connection refused" 2022-12-03T01:53:44.454 INFO:teuthology.orchestra.run.smithi155.stderr:: exit status 1] 2022-12-03T01:53:44.455 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:53:44.456 INFO:teuthology.orchestra.run.smithi155.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet] 2022-12-03T01:53:44.456 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:53:44.456 INFO:teuthology.orchestra.run.smithi155.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:53:44.457 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:53:44.457 INFO:teuthology.orchestra.run.smithi155.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:53:44.458 INFO:teuthology.orchestra.run.smithi155.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:53:44.458 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:53:44.458 INFO:teuthology.orchestra.run.smithi155.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:53:44.459 INFO:teuthology.orchestra.run.smithi155.stdout:to reset your system's IPVS tables. 2022-12-03T01:53:44.459 INFO:teuthology.orchestra.run.smithi155.stdout: 2022-12-03T01:53:44.459 INFO:teuthology.orchestra.run.smithi155.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:53:44.460 INFO:teuthology.orchestra.run.smithi155.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:53:44.461 INFO:tasks.kubeadm:Uninstalling kube{adm,let,ctl} 2022-12-03T01:53:44.462 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-12-03T01:53:44.474 INFO:teuthology.task.clock:Checking final clock skew... 2022-12-03T01:53:44.474 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:53:44.478 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:53:44.481 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:53:44.499 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:53:44.500 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2022-12-03T01:53:44.500 INFO:teuthology.orchestra.run.smithi137.stdout:+hv01.front.sepi 216.240.36.24 3 u 61 64 377 0.096 3.026 0.279 2022-12-03T01:53:44.501 INFO:teuthology.orchestra.run.smithi137.stdout:*hv02.front.sepi 163.237.218.18 2 u 62 64 377 0.086 1.737 0.297 2022-12-03T01:53:44.501 INFO:teuthology.orchestra.run.smithi137.stdout:+hv03.front.sepi 206.82.28.3 3 u 52 64 377 0.099 1.757 0.263 2022-12-03T01:53:44.501 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:53:44.504 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:53:44.504 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2022-12-03T01:53:44.505 INFO:teuthology.orchestra.run.smithi142.stdout:+hv01.front.sepi 216.240.36.24 3 u 1 64 377 0.109 -0.498 0.177 2022-12-03T01:53:44.505 INFO:teuthology.orchestra.run.smithi142.stdout:*hv02.front.sepi 163.237.218.18 2 u 4 64 377 0.092 -1.070 0.330 2022-12-03T01:53:44.505 INFO:teuthology.orchestra.run.smithi142.stdout:+hv03.front.sepi 206.82.28.3 3 u 2 64 377 0.090 -1.182 0.261 2022-12-03T01:53:44.506 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:53:44.522 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:53:44.523 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2022-12-03T01:53:44.523 INFO:teuthology.orchestra.run.smithi155.stdout:+hv01.front.sepi 216.240.36.24 3 u 13 64 377 0.098 0.085 0.275 2022-12-03T01:53:44.523 INFO:teuthology.orchestra.run.smithi155.stdout:*hv02.front.sepi 163.237.218.18 2 u 5 64 377 0.097 -0.280 0.407 2022-12-03T01:53:44.524 INFO:teuthology.orchestra.run.smithi155.stdout:+hv03.front.sepi 206.82.28.3 3 u - 64 377 0.108 -0.813 0.260 2022-12-03T01:53:44.524 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:53:44.525 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-12-03T01:53:44.541 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-12-03T01:53:44.542 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-12-03T01:53:44.578 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-12-03T01:53:44.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-12-03T01:53:44.667 INFO:teuthology.task.internal:Duration was 1660.730066 seconds 2022-12-03T01:53:44.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-12-03T01:53:44.711 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-12-03T01:53:44.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:53:44.715 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:53:44.718 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:53:44.764 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-12-03T01:53:44.765 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2022-12-03T01:53:44.766 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:53:44.780 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2022-12-03T01:53:44.781 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:53:44.819 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2022-12-03T01:53:44.820 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:53:44.835 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T01:53:44.836 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:44.839 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:44.863 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:53:44.931 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-12-03T01:53:44.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:53:44.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:53:44.950 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:53:45.055 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-12-03T01:53:45.069 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-12-03T01:53:45.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:53:45.087 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:53:45.105 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:53:45.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-12-03T01:53:45.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.150 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.158 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2022-12-03T01:53:45.161 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2022-12-03T01:53:45.179 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2022-12-03T01:53:45.206 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:45.232 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:45.250 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:53:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:53:45.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-12-03T01:53:45.275 INFO:teuthology.task.internal:Transferring archived files... 2022-12-03T01:53:45.277 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/smithi137 2022-12-03T01:53:45.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:53:45.317 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/smithi142 2022-12-03T01:53:45.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:53:45.361 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687/remote/smithi155 2022-12-03T01:53:45.362 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:53:45.411 INFO:teuthology.task.internal:Removing archive directory... 2022-12-03T01:53:45.412 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:53:45.415 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:53:45.418 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:53:45.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-12-03T01:53:45.482 INFO:teuthology.task.internal:Not uploading archives. 2022-12-03T01:53:45.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-12-03T01:53:45.535 INFO:teuthology.task.internal:Tidying up after the test... 2022-12-03T01:53:45.536 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:53:45.539 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:53:45.542 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:53:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:53 /home/ubuntu/cephtest 2022-12-03T01:53:45.547 INFO:teuthology.orchestra.run.smithi142.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:53 /home/ubuntu/cephtest 2022-12-03T01:53:45.550 INFO:teuthology.orchestra.run.smithi155.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:53 /home/ubuntu/cephtest 2022-12-03T01:53:45.552 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2022-12-03T01:53:45.645 INFO:teuthology.nuke:Checking targets against current locks 2022-12-03T01:53:45.677 DEBUG:teuthology.nuke:shortname: smithi137 2022-12-03T01:53:45.678 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:53:45.702 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.111414', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2WxWvWjADzCCGdPLzJAfBVCPIIeRZMhVKgcnENlSq4dXWPd5XnvFsx6dFcOgiuL4JN6T93GwZBuzhTI77xNlNNamDJ4IHDi6OmNlg2z56M3SmeYXNY21V/Egshvvg7FS+ZZ3MQw5g3DIM5ryyE+is5kh4nbzBtuALmr9nMp4/oTcIOUcf/EJk67r9IPdDhhv3EAqmiI+37Iqk803prTb3mpb54iUutAXtFMsm2+6OgiUlmEmJxukKCtkx9adv/1iv6eokimTY4khDueSr+0GuR5nWLdG3mPRbASSERoyQJ83r2ttaeHJB8fB7B6lFvK1c/a6kN5YBH6ZYbPr56GDAuAlbhw7Rr7ze1RLEdNtYW7k8mhWsR33EKYEhhLFiZau/dWqEbaWb1xRHL6czWZd90UOJtTxyDNdgVh61/vvl0kIib3IYPX9ZmEZuou+hZ2apgoAit94cNQ2rGu4kXCuB/OUvbJXmfpospaGYoj5J98X0djV+yQ9CKULdbCHOkCE='} 2022-12-03T01:53:45.710 DEBUG:teuthology.nuke:shortname: smithi142 2022-12-03T01:53:45.710 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:53:45.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.109816', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTYaYroROwpyrYbKS3rK2RLDVtuSNcELmQ+hu7NtGWHZlZDEEWJZEKK4MS6IFBrKUXxXMOto22yjBfBQ6AWDuoXXVGSNN6/cTCfwVGya0MuedcKvM2XNF42DrJjO1M9YX6RbHYLY4lRF5UHWUGF87L7ilrrdvJkxAPzbc+1LsLLALpb8GJQ9tOXbEuosBGImmvxQQ4yfkxkrEEjNRctnD3o9M9e/is9ZxN0WT2+OzzTmMMRAs0Fc8SJ4XfHHVFFYsOhHBjwleqiQj1m0DdnTed/yQd502sDuwvP7Mv08ACRfbbl3q2Dw+tI4XHVu7Iz08ThE93iAwoJnnuFtRxLTB2ccI0rwhlnOOQgEKQBJUV5CcXolUoukwEr2apei2gQBlMGvxwO8qjbUtFLHP/IT70GCAv5tJxIz7fOtTcriV+xU2LHMhfJsQ76l680PwNa9l7P0k54jHs8vAVxYvTZt+oat6smaVg5VHJPRT605Nr6h3zh5YjIp0s8lkYAWp8uwk='} 2022-12-03T01:53:45.742 DEBUG:teuthology.nuke:shortname: smithi155 2022-12-03T01:53:45.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:53:45.783 INFO:teuthology.orchestra.console:Power off smithi137 2022-12-03T01:53:45.785 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:53:45.805 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101687', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:16:59.112514', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjpaEPYI6dhOtNNDoVaKsI709Lv2pjzuu/WTfJBz1r9VXRwpX9+6KOvrQrWGBHysmvw5PFy8F6garVKz6DKm8xHsDYdfbIMPxR2jzKsrBY8pCMKl7/3yIJQCLP5oX/Cn4pltJyJhIsvdHRPiaCEybLuXsIs8BUcKqI82rDehvZrAb2yco4WdoAPtW+7B3CahnjQu76zX6Q1z35Vq/StCm02sGnIipqF7X6vwnY/U1DsgqH1sdCGynlhRTsz1sgbBv8m804jZMJKGWAEnFqR+OGkPLBcvP0/E4hHk6Ha3AB82odBR+kUHShR0pikcEjIfO5gRC1ZRZY3gp4KZg/C+eb4tvMx9I68BInRpSft3R4coh2ElNkFUIY6kJrAQMTllAt4cZqKDqGtrzDqxaam00K+QAqAtg+HeKaHK4Tv/QwZWgX+JLlJpQak1zuAEXMWBf1efMse3r082zm03Dxtnv5JNI80eXAkH/ocK5DJv3JJ64XVO3evo9IbI+MlUoFR2E='} 2022-12-03T01:53:45.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:45.851 INFO:teuthology.orchestra.console:Power off smithi142 2022-12-03T01:53:45.852 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:53:45.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:45.908 INFO:teuthology.orchestra.console:Power off smithi155 2022-12-03T01:53:45.909 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:53:45.943 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:49.873 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:49.924 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:49.977 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:53.995 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:54.038 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:54.093 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:54.309 INFO:teuthology.orchestra.console:Power off for smithi155 completed 2022-12-03T01:53:58.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:58.152 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:53:58.329 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2022-12-03T01:53:58.368 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2022-12-03T01:53:58.474 INFO:teuthology.run:Summary data: description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/master} duration: 1660.7300658226013 failure_reason: '''check osd count'' reached maximum tries (90) after waiting for 900 seconds' owner: scheduled_lflores@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=338664706450452e848acdede7af5d19 status: fail success: false 2022-12-03T01:53:58.475 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-12-03T01:53:58.551 INFO:teuthology.run:FAIL