2022-12-03T01:16:43.812 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-12-03T01:16:43.815 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682 --name lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} --owner scheduled_lflores@teuthology --verbose -- /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/orig.config.yaml 2022-12-03T01:16:43.873 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-12-03T01:16:43.951 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682 branch: wip-yuri7-testing-2022-11-18-1548 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} email: null first_in_suite: false job_id: '7101682' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri7-testing-2022-11-18-1548 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri7-testing-2022-11-18-1548 sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri7-testing-2022-11-18-1548 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 targets: smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEY+i4wrK2zvejhB63LailoMq8pVSff2L0VUWp4HU07jR0KeoghN2mxvuvxPb3VyJ5FopueYw7HBahS+oiPLNozJU8PtNTmC1WPHLDUZ6UlAkBGhNevPhTWhjE43PsYQZRV4wRYvDIiEtvBJhydOl6GB+6MP5KKiIL3ySYchpkCxdXVhxjHqMaDe/P8UOTlWmaM6MEuQfizBRDbvFQyfow/1hJ7iek2v4NQTk130hvCjIKEJzyRSXWG4KPg54mpTomWTCxKOBBBtLVrXHQo+7l1TzTOdXcCIfZeux15uoNBDFNT2MeCMqTf6jzfannR0pJwPwQ6cD/USvejLEh4s3ySjjX0WCJQMnbKMQy3qAwNjcrxAPu+wP9d/A520rrroJNURgI+z4X8bzifCpjZ2l0zRmJ/HqEzJyaqcRF3Ldh1SEElUNVb2ukB83DJudofvmu8ypG/BBbLKx/RTNrFEBoySlZs4FNwm9SL1AbIrLCRi6A4mBVn6/riSvCGCZySf8= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIvh7ryYHiaQh2Dqw+PCcM93p9ca0Rf/8CP4kp7w9y0oJ6/XIH/LXXXWDl4vT1LKHo/7YJCUq9l0xy0+oAjvuNYMJJEtbou64msidOTPLaEwkbLlbwopnNKCp5MrcHQ9Jsi8O0XXMyqMgZbjaUHeMhARdK/H3CVc1hlxtpgl2D+7BgNHu20/AwAI0EzqluDKlF7ghhCJOAnIt4SACBe777/yYYSDdVrsVHh4NEDBahtM7MVd3gsO0lZOJaE+eCh5cQ+ieaZ1T1igpV4b3AzuJyivBq1UdcIMRyJ9dMHzp7DvahSLwT8awr8LtHFyip7PZlifq7r1KqJIedb6oH8Xr7+TVYCBLblvJxyG7Q0qos9pfzQFQ2Yri6g8mm2d6f2RcgXQatyaoaoKh2IBkikwL81TqotZSocGex0ze7LcnNaeLSoDaGPOb2/mvvB16sMYBu1fO+Eyc4O8NSnuidKg9sg79wEZC7akoLgLmCLgLvZ9w0P42Z9MBQ2s1uWjg4ths= smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/hO+jEDXI7jJAhwTvzKqeELuuMek2Mj/hs6PWGfAWgX0xYAIRkediizPkh8kfDtWym2QannL8tZjtgmC1thvZ4+eHF6Zx6Gv4vAfh4uONdV74Iql6lwTLCS+q+kQ7ZXBvn4pNQj+/YCgGhb6O61pQ3Bbjy0NDJYhWQ4rpvF3HhKoYdhkmDXzAU6iDLQ8n4Bh6nV1K9icCLsYcnQi+roxTIT+QMIPDWimXR7yaRvd8ZNTv6A+C+zGeQu8gzFNBSkWWqmFoORqnLu2GKW6JlxSWEFT1IIAZUc0zg8Tig+N6qg33KwJmU46QP0x7wbIACqrId9n8YiIlqhPIU3z5YG5KIqaeD6QS9gZNQZPI6zDQSSTLOkO45l73SORGGGBxZ/gojMyF94mfD7Uj7klpeZid6MpKbs0OXCvZgVAKsFbPvy7FV9tODC6LUG79ArL1C65h9DT22zUYKff2+wKDQpYV/5tBJyVMPd/Edx6tT7mkRyMeIL5aV9rkkTk4B6C1bY8= tasks: - kubeadm: null - nvme_loop: null - rook: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4da97cf64e542f347ec47b7bdbe5eca99759f9b7 timestamp: 2022-12-02_15:43:15 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23126 2022-12-03T01:16:43.952 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa; will attempt to use it 2022-12-03T01:16:43.953 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks 2022-12-03T01:16:43.954 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-12-03T01:16:43.957 INFO:teuthology.task.internal:Checking packages... 2022-12-03T01:16:43.990 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '48d203d9b836cc224eb4085f6198d948c6fa1bb1' 2022-12-03T01:16:43.990 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-12-03T01:16:43.991 INFO:teuthology.packaging:ref: None 2022-12-03T01:16:43.991 INFO:teuthology.packaging:tag: None 2022-12-03T01:16:43.992 INFO:teuthology.packaging:branch: wip-yuri7-testing-2022-11-18-1548 2022-12-03T01:16:43.993 INFO:teuthology.packaging:sha1: 48d203d9b836cc224eb4085f6198d948c6fa1bb1 2022-12-03T01:16:43.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri7-testing-2022-11-18-1548 2022-12-03T01:16:44.168 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-872-g48d203d9-1focal 2022-12-03T01:16:44.169 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-12-03T01:16:44.182 INFO:teuthology.task.internal:no buildpackages task found 2022-12-03T01:16:44.183 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-12-03T01:16:44.229 INFO:teuthology.task.internal:Saving configuration 2022-12-03T01:16:44.255 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-12-03T01:16:44.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:16:44.317 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.573656', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINu/0TtfDSlwLDgHi2G8ibZ0S3s050cVeE8UGW95QeBK'} 2022-12-03T01:16:44.343 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.572369', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICwK8qHTZAnLxdOqugMy2D15sRDkB/VGpYqxn5OrkPRD'} 2022-12-03T01:16:44.371 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.570268', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINXm3hJyx1BHh/+Od5z8kXfJ+RbAiCbxHE52az91AFTp'} 2022-12-03T01:16:44.372 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-12-03T01:16:44.379 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-12-03T01:16:44.380 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-12-03T01:16:44.380 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-12-03T01:16:44.381 INFO:teuthology.run_tasks:Running task console_log... 2022-12-03T01:16:44.530 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdaa72b92f0>, signals=[15]) 2022-12-03T01:16:44.532 INFO:teuthology.run_tasks:Running task internal.connect... 2022-12-03T01:16:44.539 INFO:teuthology.task.internal:Opening connections... 2022-12-03T01:16:44.539 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2022-12-03T01:16:44.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:16:44.682 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2022-12-03T01:16:44.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:16:44.807 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2022-12-03T01:16:44.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:16:44.913 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-12-03T01:16:44.923 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2022-12-03T01:16:44.933 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2022-12-03T01:16:44.934 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2022-12-03T01:16:44.984 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2022-12-03T01:16:44.985 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:16:44.985 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2022-12-03T01:16:44.986 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2022-12-03T01:16:44.986 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:16:44.987 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="20.04" 2022-12-03T01:16:44.987 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:16:44.988 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:16:44.988 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:16:44.988 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:16:44.989 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=focal 2022-12-03T01:16:44.989 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:16:44.991 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2022-12-03T01:16:45.045 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2022-12-03T01:16:45.059 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2022-12-03T01:16:45.060 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2022-12-03T01:16:45.110 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2022-12-03T01:16:45.112 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:16:45.112 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2022-12-03T01:16:45.113 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2022-12-03T01:16:45.114 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:16:45.114 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="20.04" 2022-12-03T01:16:45.116 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:16:45.116 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:16:45.117 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:16:45.117 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:16:45.117 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=focal 2022-12-03T01:16:45.118 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:16:45.119 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-12-03T01:16:45.145 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2022-12-03T01:16:45.155 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2022-12-03T01:16:45.156 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2022-12-03T01:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Ubuntu" 2022-12-03T01:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="20.04.5 LTS (Focal Fossa)" 2022-12-03T01:16:45.203 INFO:teuthology.orchestra.run.smithi138.stdout:ID=ubuntu 2022-12-03T01:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE=debian 2022-12-03T01:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Ubuntu 20.04.5 LTS" 2022-12-03T01:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="20.04" 2022-12-03T01:16:45.204 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.ubuntu.com/" 2022-12-03T01:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-12-03T01:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-12-03T01:16:45.205 INFO:teuthology.orchestra.run.smithi138.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-12-03T01:16:45.206 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_CODENAME=focal 2022-12-03T01:16:45.206 INFO:teuthology.orchestra.run.smithi138.stdout:UBUNTU_CODENAME=focal 2022-12-03T01:16:45.206 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2022-12-03T01:16:45.230 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-12-03T01:16:45.245 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-12-03T01:16:45.287 INFO:teuthology.task.internal:Checking for old test directory... 2022-12-03T01:16:45.288 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:16:45.291 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:16:45.293 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2022-12-03T01:16:45.300 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-12-03T01:16:45.312 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-12-03T01:16:45.312 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:16:45.321 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:16:45.346 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2022-12-03T01:16:45.358 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-12-03T01:16:45.465 INFO:teuthology.run_tasks:Running task kernel... 2022-12-03T01:16:45.496 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-12-03T01:16:45.497 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-12-03T01:16:45.502 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-12-03T01:16:45.503 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-12-03T01:16:45.504 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:16:45.507 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:16:45.508 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2022-12-03T01:16:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:16:45.527 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2022-12-03T01:16:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:16:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:16:45.530 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2022-12-03T01:16:45.530 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-12-03T01:16:45.541 INFO:teuthology.orchestra.run.smithi138.stdout:5.4.0-128-generic 2022-12-03T01:16:45.543 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.4.0-128-generic 2022-12-03T01:16:45.543 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2022-12-03T01:16:45.551 INFO:teuthology.orchestra.run.smithi092.stdout:5.4.0-128-generic 2022-12-03T01:16:45.552 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.4.0-128-generic 2022-12-03T01:16:45.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2022-12-03T01:16:45.590 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-128-generic 2022-12-03T01:16:45.591 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.4.0-128-generic 2022-12-03T01:16:45.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-12-03T01:16:45.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2022-12-03T01:16:45.705 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2022-12-03T01:16:45.713 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-12-03T01:16:45.853 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:16:45.860 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:16:45.861 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:16:45.861 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:16:45.934 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:16:45.958 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:16:46.001 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:16:46.002 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-12-03T01:16:46.040 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:16:46.049 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-12-03T01:16:46.136 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:16:46.227 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:16:46.232 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:16:46.270 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:16:46.275 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:16:46.275 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:16:46.291 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:16:46.291 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:16:46.292 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:16:46.292 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:16:46.296 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:16:46.304 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:16:46.304 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:16:46.318 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:16:46.324 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:16:46.324 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:16:46.325 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:16:46.334 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:16:46.369 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:16:46.384 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:16:46.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:16:46.386 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:16:46.387 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:16:46.387 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:16:46.399 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-12-03T01:16:46.400 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:16:46.401 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:16:46.401 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:16:46.411 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:16:46.411 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:16:46.413 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:16:46.425 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:16:46.426 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:16:46.426 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:16:46.426 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:16:46.435 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:16:46.475 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [530 kB] 2022-12-03T01:16:46.475 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:16:46.480 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:16:46.481 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:16:46.492 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:16:46.493 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:16:46.494 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:46.494 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:16:46.499 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:16:46.507 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:16:46.508 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:16:46.512 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:16:46.512 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:16:46.513 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:16:46.513 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:46.514 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:16:46.514 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:16:46.527 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:16:46.532 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:16:46.532 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:16:46.533 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:16:46.533 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:16:46.548 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:16:46.612 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:16:46.787 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,889 kB] 2022-12-03T01:16:46.873 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [761 kB] 2022-12-03T01:16:46.907 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:16:46.919 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:16:46.920 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:16:46.921 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,263 kB] 2022-12-03T01:16:46.962 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:16:46.963 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:16:46.965 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [310 kB] 2022-12-03T01:16:46.967 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:46.967 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:16:46.978 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.4 kB] 2022-12-03T01:16:46.978 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [27.9 kB] 2022-12-03T01:16:46.979 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,383 kB] 2022-12-03T01:16:46.990 INFO:teuthology.orchestra.run.smithi092.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:16:47.009 INFO:teuthology.orchestra.run.smithi092.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:16:47.038 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [195 kB] 2022-12-03T01:16:47.039 INFO:teuthology.orchestra.run.smithi092.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:16:47.040 INFO:teuthology.orchestra.run.smithi092.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:16:47.040 INFO:teuthology.orchestra.run.smithi092.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:16:47.043 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [596 B] 2022-12-03T01:16:47.044 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [573 kB] 2022-12-03T01:16:47.057 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [777 kB] 2022-12-03T01:16:47.074 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [149 kB] 2022-12-03T01:16:47.076 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.8 kB] 2022-12-03T01:16:47.086 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [394 kB] 2022-12-03T01:16:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:16:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:47.092 INFO:teuthology.orchestra.run.smithi092.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:16:47.096 INFO:teuthology.orchestra.run.smithi092.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:16:47.098 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.1 kB] 2022-12-03T01:16:47.100 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [29.3 kB] 2022-12-03T01:16:47.101 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,470 kB] 2022-12-03T01:16:47.120 INFO:teuthology.orchestra.run.smithi092.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:16:47.120 INFO:teuthology.orchestra.run.smithi092.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:16:47.121 INFO:teuthology.orchestra.run.smithi092.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:16:47.131 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-12-03T01:16:47.131 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,184 B] 2022-12-03T01:16:47.138 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,464 B] 2022-12-03T01:16:47.139 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [516 B] 2022-12-03T01:16:47.148 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [208 kB] 2022-12-03T01:16:47.154 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:47.155 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,007 kB] 2022-12-03T01:16:47.182 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [704 kB] 2022-12-03T01:16:47.195 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [234 kB] 2022-12-03T01:16:47.198 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2022-12-03T01:16:47.199 INFO:teuthology.orchestra.run.smithi092.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:16:47.200 INFO:teuthology.orchestra.run.smithi092.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:16:47.242 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,476 B] 2022-12-03T01:16:47.243 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2022-12-03T01:16:47.243 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,380 B] 2022-12-03T01:16:47.244 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-12-03T01:16:47.244 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2022-12-03T01:16:47.249 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2022-12-03T01:16:47.334 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-12-03T01:16:47.334 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2022-12-03T01:16:47.335 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2022-12-03T01:16:47.336 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.3 kB] 2022-12-03T01:16:47.336 INFO:teuthology.orchestra.run.smithi138.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2022-12-03T01:16:50.966 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 13.5 MB in 2s (6,961 kB/s) 2022-12-03T01:16:51.127 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 13.5 MB in 2s (6,655 kB/s) 2022-12-03T01:16:51.592 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 13.5 MB in 3s (5,273 kB/s) 2022-12-03T01:16:52.176 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:16:52.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:16:52.270 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:16:52.451 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:16:52.454 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:16:52.454 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:16:52.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:16:52.548 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:16:52.603 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2022-12-03T01:16:52.603 INFO:teuthology.orchestra.run.smithi092.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:52.603 INFO:teuthology.orchestra.run.smithi092.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:16:52.604 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:16:52.604 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2022-12-03T01:16:52.605 INFO:teuthology.orchestra.run.smithi092.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:16:52.605 INFO:teuthology.orchestra.run.smithi092.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:16:52.659 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-12-03T01:16:52.660 INFO:teuthology.orchestra.run.smithi092.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:52.660 INFO:teuthology.orchestra.run.smithi092.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:16:52.661 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:16:52.661 INFO:teuthology.orchestra.run.smithi092.stdout: thermald upower usbmuxd 2022-12-03T01:16:52.730 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:16:52.731 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:16:52.836 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:16:52.836 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:16:52.836 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:16:52.837 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:16:52.877 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-12-03T01:16:52.877 INFO:teuthology.orchestra.run.smithi121.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:52.877 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:16:52.878 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:16:52.878 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-12-03T01:16:52.878 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:16:52.879 INFO:teuthology.orchestra.run.smithi121.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:16:52.925 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:16:52.936 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-12-03T01:16:52.936 INFO:teuthology.orchestra.run.smithi121.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:52.936 INFO:teuthology.orchestra.run.smithi121.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:16:52.937 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:16:52.937 INFO:teuthology.orchestra.run.smithi121.stdout: thermald upower usbmuxd 2022-12-03T01:16:52.952 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:16:52.990 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:16:52.991 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:16:52.991 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:16:52.991 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:16:53.034 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:16:53.052 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:16:53.058 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:16:53.060 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:16:53.075 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:16:53.081 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:16:53.102 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:16:53.132 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:16:53.148 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:16:53.216 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:16:53.217 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:16:53.222 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:16:53.255 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:16:53.379 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2022-12-03T01:16:53.382 INFO:teuthology.orchestra.run.smithi138.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:53.383 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-modules-5.15.0-56-generic 2022-12-03T01:16:53.383 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-extra-5.15.0-56-generic thermald upower usbmuxd 2022-12-03T01:16:53.385 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:16:53.385 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2022-12-03T01:16:53.386 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-12-03T01:16:53.386 INFO:teuthology.orchestra.run.smithi138.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-56-generic 2022-12-03T01:16:53.427 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2022-12-03T01:16:53.428 INFO:teuthology.orchestra.run.smithi138.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-12-03T01:16:53.429 INFO:teuthology.orchestra.run.smithi138.stdout: libusbmuxd6 linux-image-5.15.0-56-generic linux-image-generic-hwe-20.04 2022-12-03T01:16:53.429 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-5.15.0-56-generic linux-modules-extra-5.15.0-56-generic 2022-12-03T01:16:53.429 INFO:teuthology.orchestra.run.smithi138.stdout: thermald upower usbmuxd 2022-12-03T01:16:53.480 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:16:53.610 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 13 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:16:53.610 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 90.3 MB of archives. 2022-12-03T01:16:53.614 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 485 MB of additional disk space will be used. 2022-12-03T01:16:53.615 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-12-03T01:16:53.900 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-12-03T01:16:53.917 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-12-03T01:16:53.929 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-12-03T01:16:54.001 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-12-03T01:16:54.037 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [20.6 MB] 2022-12-03T01:16:54.147 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:16:54.226 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:16:54.226 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:16:54.227 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:16:54.230 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:16:54.240 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:16:54.437 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:16:54.635 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 90.3 MB in 1s (70.1 MB/s) 2022-12-03T01:16:54.836 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:16:54.897 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [11.3 MB] 2022-12-03T01:16:55.179 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-56-generic amd64 5.15.0-56.62~20.04.1 [57.9 MB] 2022-12-03T01:16:56.188 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:16:56.193 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:16:56.226 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:16:56.302 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:16:56.303 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:16:56.312 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:16:56.313 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:16:56.352 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:16:56.713 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 90.3 MB in 4s (24.8 MB/s) 2022-12-03T01:16:56.724 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:16:56.734 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:16:56.780 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:16:56.875 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:16:57.020 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.56.62~20.04.22 [2,832 B] 2022-12-03T01:16:57.100 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-12-03T01:16:57.102 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-12-03T01:16:57.112 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-12-03T01:16:57.112 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-12-03T01:16:57.131 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:16:57.147 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:16:57.189 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:16:57.467 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:16:57.480 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:16:57.509 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 90.3 MB in 4s (24.6 MB/s) 2022-12-03T01:16:57.534 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:16:57.731 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-12-03T01:16:57.853 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:16:57.868 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:16:57.911 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:16:58.189 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:16:58.204 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:16:58.241 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:16:58.342 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:16:58.347 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:16:58.600 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:16:58.891 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:16:58.907 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:16:58.941 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:16:59.131 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102417 files and directories currently installed.) 2022-12-03T01:16:59.134 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-12-03T01:16:59.269 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:16:59.291 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:16:59.299 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:16:59.335 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:16:59.592 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libplist3:amd64. 2022-12-03T01:16:59.604 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-12-03T01:16:59.648 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:16:59.698 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:16:59.714 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:16:59.755 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:16:59.996 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-12-03T01:17:00.012 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-12-03T01:17:00.041 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:17:00.051 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:17:00.060 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:17:00.091 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:17:00.395 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:17:00.398 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-12-03T01:17:00.407 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:00.408 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-12-03T01:17:00.446 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:00.456 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:17:00.653 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:17:00.663 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:00.753 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:00.757 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-12-03T01:17:00.771 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-12-03T01:17:00.824 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:17:01.151 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-5.15.0-56-generic. 2022-12-03T01:17:01.156 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../05-linux-modules-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:01.208 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:01.247 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:17:01.260 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:01.305 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:02.953 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:17:02.969 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:03.048 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:03.548 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:17:03.556 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:03.594 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:03.602 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-5.15.0-56-generic. 2022-12-03T01:17:03.614 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../06-linux-image-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:03.703 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:04.255 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-56-generic. 2022-12-03T01:17:04.269 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-56-generic_5.15.0-56.62~20.04.1_amd64.deb ... 2022-12-03T01:17:04.313 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:08.011 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:17:08.027 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:17:08.062 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:08.379 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:17:08.390 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:17:08.422 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:08.809 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package thermald. 2022-12-03T01:17:08.824 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:17:08.867 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:09.414 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package upower. 2022-12-03T01:17:09.436 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:17:09.481 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:17:09.910 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:17:09.923 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:17:09.959 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:10.267 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:17:10.363 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:17:10.468 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:17:10.481 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:17:10.484 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:17:10.527 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:10.624 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:17:10.750 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:17:10.887 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:17:10.888 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:10.894 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:17:10.942 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:11.160 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:17:11.182 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-12-03T01:17:11.199 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.56.62~20.04.22_amd64.deb ... 2022-12-03T01:17:11.231 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:11.341 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package thermald. 2022-12-03T01:17:11.355 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:17:11.408 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:11.546 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libevdev2:amd64. 2022-12-03T01:17:11.560 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-12-03T01:17:11.603 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:11.954 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package upower. 2022-12-03T01:17:11.955 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package thermald. 2022-12-03T01:17:11.967 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:17:11.970 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-12-03T01:17:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:12.006 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:12.014 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:17:12.031 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:12.209 INFO:teuthology.orchestra.run.smithi121.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:17:12.213 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:17:12.214 INFO:teuthology.orchestra.run.smithi121.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:17:12.393 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:17:12.408 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:17:12.441 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:12.544 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package upower. 2022-12-03T01:17:12.558 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-12-03T01:17:12.629 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking upower (0.99.11-1build2) ... 2022-12-03T01:17:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:17:12.788 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:17:12.937 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:17:12.945 INFO:teuthology.orchestra.run.smithi121.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:12.983 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:13.023 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package usbmuxd. 2022-12-03T01:17:13.037 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-12-03T01:17:13.063 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:17:13.189 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:17:13.200 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:13.309 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:17:13.310 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:13.310 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:13.418 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:13.520 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:17:13.755 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-12-03T01:17:13.756 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:13.859 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-12-03T01:17:13.968 INFO:teuthology.orchestra.run.smithi092.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:13.979 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-12-03T01:17:14.012 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:14.080 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-12-03T01:17:14.188 INFO:teuthology.orchestra.run.smithi092.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:17:14.189 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:17:14.189 INFO:teuthology.orchestra.run.smithi092.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:17:14.208 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-12-03T01:17:14.351 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-12-03T01:17:14.477 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up upower (0.99.11-1build2) ... 2022-12-03T01:17:14.685 INFO:teuthology.orchestra.run.smithi092.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:17:14.903 INFO:teuthology.orchestra.run.smithi092.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:14.905 INFO:teuthology.orchestra.run.smithi138.stdout:upower.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:14.953 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-12-03T01:17:15.004 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:15.147 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:15.152 INFO:teuthology.orchestra.run.smithi138.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-12-03T01:17:15.165 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user `usbmux' (UID 114) ... 2022-12-03T01:17:15.166 INFO:teuthology.orchestra.run.smithi138.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-12-03T01:17:15.319 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:15.319 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:15.652 INFO:teuthology.orchestra.run.smithi138.stdout:Not creating home directory `/var/lib/usbmux'. 2022-12-03T01:17:15.762 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:15.840 INFO:teuthology.orchestra.run.smithi138.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-12-03T01:17:15.882 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-12-03T01:17:16.199 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:16.199 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-12-03T01:17:16.490 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:16.490 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:16.632 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:16.661 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:17.188 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:17.984 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:18.091 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:18.139 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:17:18.993 INFO:teuthology.orchestra.run.smithi092.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:18.994 INFO:teuthology.orchestra.run.smithi092.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:19.120 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:19.633 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:19.634 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:19.750 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-extra-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:20.511 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:20.634 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:17:21.180 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.56.62~20.04.22) ... 2022-12-03T01:17:21.294 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:17:21.782 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:17:23.136 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:17:23.403 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:23.473 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:17:23.473 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:24.928 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:17:25.525 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:17:25.618 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:25.693 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:17:25.694 INFO:teuthology.orchestra.run.smithi092.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:25.734 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:17:26.323 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-12-03T01:17:26.422 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for linux-image-5.15.0-56-generic (5.15.0-56.62~20.04.1) ... 2022-12-03T01:17:26.496 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-12-03T01:17:26.496 INFO:teuthology.orchestra.run.smithi138.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:48.267 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:17:48.268 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:17:48.276 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:48.284 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2022-12-03T01:17:48.611 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:48.639 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:49.036 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:49.047 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:17:50.616 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:17:50.616 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:17:50.653 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:50.654 INFO:teuthology.orchestra.run.smithi092.stdout:Generating grub configuration file ... 2022-12-03T01:17:50.991 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:51.027 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:51.107 INFO:teuthology.orchestra.run.smithi121.stdout:done 2022-12-03T01:17:51.408 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:51.413 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:17:51.458 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:51.492 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-12-03T01:17:51.496 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub' 2022-12-03T01:17:51.498 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:51.498 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-12-03T01:17:51.498 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-12-03T01:17:51.499 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-12-03T01:17:51.500 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 20 2022-12-03T01:17:51.500 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:51.500 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-12-03T01:17:51.500 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:51.501 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:51.501 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:51.501 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:51.502 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-12-03T01:17:51.502 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:51.502 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:51.505 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-12-03T01:17:51.505 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:17:51.506 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:17:51.506 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:17:51.507 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-12-03T01:17:51.508 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:17:51.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-12-03T01:17:51.519 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:51.519 INFO:teuthology.orchestra.run.smithi138.stdout:Generating grub configuration file ... 2022-12-03T01:17:51.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-12-03T01:17:51.768 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:17:51.851 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:51.875 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:51.879 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:17:51.959 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:17:52.039 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:17:52.373 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:52.383 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:17:53.386 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:17:53.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:17:53.435 INFO:teuthology.orchestra.run.smithi092.stdout:done 2022-12-03T01:17:53.497 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:17:53.688 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:17:53.688 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:17:53.823 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:53.864 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:53.864 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2022-12-03T01:17:53.865 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2022-12-03T01:17:53.873 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2022-12-03T01:17:53.874 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 20 2022-12-03T01:17:53.874 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:53.874 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2022-12-03T01:17:53.875 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:53.875 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:53.882 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:53.883 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:53.883 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2022-12-03T01:17:53.884 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:53.884 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:53.885 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2022-12-03T01:17:53.885 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:17:53.886 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:17:53.888 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:17:53.888 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-12-03T01:17:53.889 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:17:53.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2022-12-03T01:17:53.891 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:17:53.891 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:17:53.897 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:53.926 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:53.927 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-12-03T01:17:53.927 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-12-03T01:17:53.927 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-12-03T01:17:53.927 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 20 2022-12-03T01:17:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-12-03T01:17:53.928 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:53.929 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:53.929 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:53.929 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:53.930 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-12-03T01:17:53.930 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:53.930 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:53.930 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-12-03T01:17:53.932 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2022-12-03T01:17:53.947 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.aZw5WIbizo 2022-12-03T01:17:53.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.aZw5WIbizo 2022-12-03T01:17:53.998 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt-get update 2022-12-03T01:17:54.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.aZw5WIbizo 2022-12-03T01:17:54.098 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.aZw5WIbizo is 9KB 2022-12-03T01:17:54.157 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.aZw5WIbizo 2022-12-03T01:17:54.181 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:54.230 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:17:54.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:17:54.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:54.254 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:17:54.255 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:17:54.267 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:17:54.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2022-12-03T01:17:54.326 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:17:54.389 INFO:teuthology.orchestra.run.smithi138.stdout:done 2022-12-03T01:17:54.412 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:17:54.787 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:54.821 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:54.821 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2022-12-03T01:17:54.821 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2022-12-03T01:17:54.822 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2022-12-03T01:17:54.822 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2022-12-03T01:17:54.822 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:54.822 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2022-12-03T01:17:54.823 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:54.823 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:54.823 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:54.824 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:54.824 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2022-12-03T01:17:54.824 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:54.824 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:54.825 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2022-12-03T01:17:54.825 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-128-generic Expected: 5.15.0-56-generic 2022-12-03T01:17:54.825 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-12-03T01:17:54.826 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-12-03T01:17:54.826 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-12-03T01:17:54.826 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2022-12-03T01:17:54.827 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2022-12-03T01:17:54.957 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2022-12-03T01:17:55.101 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:17:55.214 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:17:55.311 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:17:55.383 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:17:55.572 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:17:55.578 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:55.579 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2022-12-03T01:17:55.833 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:55.845 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:55.890 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:17:55.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:17:55.984 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:17:56.165 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:17:56.166 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:17:56.215 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:56.227 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:17:56.351 INFO:teuthology.orchestra.run.smithi092.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:17:56.352 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:17:56.355 DEBUG:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:56.385 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:56.386 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2022-12-03T01:17:56.387 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2022-12-03T01:17:56.388 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2022-12-03T01:17:56.388 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 20 2022-12-03T01:17:56.389 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:56.389 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2022-12-03T01:17:56.390 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:56.390 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:56.390 INFO:teuthology.orchestra.run.smithi092.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:56.391 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:56.391 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2022-12-03T01:17:56.395 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:56.396 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:56.396 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2022-12-03T01:17:56.397 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2022-12-03T01:17:56.443 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.AE50zhnDgI 2022-12-03T01:17:56.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AE50zhnDgI 2022-12-03T01:17:56.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.AE50zhnDgI 2022-12-03T01:17:56.569 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.AE50zhnDgI is 9KB 2022-12-03T01:17:56.661 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.AE50zhnDgI 2022-12-03T01:17:56.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:56.723 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:17:56.728 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:17:56.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:17:56.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:56.744 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-12-03T01:17:56.764 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:17:56.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo update-grub 2022-12-03T01:17:56.817 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:17:56.986 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:17:56.987 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:17:57.163 INFO:teuthology.orchestra.run.smithi138.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.56.62~20.04.22). 2022-12-03T01:17:57.164 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded. 2022-12-03T01:17:57.164 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic-hwe-20.04 2022-12-03T01:17:57.187 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic-hwe-20.04 2022-12-03T01:17:57.188 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2022-12-03T01:17:57.188 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2022-12-03T01:17:57.196 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2022-12-03T01:17:57.197 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2022-12-03T01:17:57.197 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2022-12-03T01:17:57.198 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2022-12-03T01:17:57.198 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta-hwe-5.15 2022-12-03T01:17:57.199 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.56.62~20.04.22 2022-12-03T01:17:57.199 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.4-0ubuntu0.1), v4l2loopback-modules (= 0.12.5-1ubuntu5), virtualbox-guest-modules (= 5.15.0-56), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-12-03T01:17:57.202 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-56-generic, linux-modules-extra-5.15.0-56-generic, linux-firmware, intel-microcode, amd64-microcode 2022-12-03T01:17:57.202 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2022-12-03T01:17:57.202 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2022-12-03T01:17:57.203 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2022-12-03T01:17:57.204 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2022-12-03T01:17:57.204 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2022-12-03T01:17:57.223 INFO:teuthology.orchestra.run.smithi121.stderr:done 2022-12-03T01:17:57.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2022-12-03T01:17:57.226 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.JSlM1U8Gma 2022-12-03T01:17:57.228 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub/grub.cfg /tmp/tmp.JSlM1U8Gma 2022-12-03T01:17:57.288 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.JSlM1U8Gma 2022-12-03T01:17:57.381 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.JSlM1U8Gma is 9KB 2022-12-03T01:17:57.441 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.JSlM1U8Gma 2022-12-03T01:17:57.456 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:57.513 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:17:57.514 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-12-03T01:17:57.514 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-12-03T01:17:57.579 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-56-generic 2022-12-03T01:17:57.580 DEBUG:teuthology.orchestra.run.smithi138:> sudo update-grub 2022-12-03T01:17:57.900 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:17:57.901 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:57.911 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2022-12-03T01:17:58.143 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:58.155 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:58.543 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:58.556 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:17:58.814 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub' 2022-12-03T01:17:58.824 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-12-03T01:17:58.825 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2022-12-03T01:17:59.181 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2022-12-03T01:17:59.197 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2022-12-03T01:17:59.559 INFO:teuthology.orchestra.run.smithi092.stderr:done 2022-12-03T01:17:59.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2022-12-03T01:17:59.649 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.4.0-128-generic 2022-12-03T01:17:59.653 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.4.0-128-generic 2022-12-03T01:18:00.627 INFO:teuthology.orchestra.run.smithi138.stderr:done 2022-12-03T01:18:00.627 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2022-12-03T01:18:27.259 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-12-03T01:18:27.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-12-03T01:18:27.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:18:29.572 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-12-03T01:18:29.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-12-03T01:18:29.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:18:30.647 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-12-03T01:18:30.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-12-03T01:18:30.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:19:27.324 DEBUG:teuthology.orchestra.remote:timed out 2022-12-03T01:19:29.578 DEBUG:teuthology.orchestra.remote:timed out 2022-12-03T01:19:30.653 DEBUG:teuthology.orchestra.remote:timed out 2022-12-03T01:19:57.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-12-03T01:19:57.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:19:57.624 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-12-03T01:19:58.111 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-12-03T01:19:58.112 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-56-generic"... 2022-12-03T01:19:58.113 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-12-03T01:19:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:5.15.0-56-generic 2022-12-03T01:19:58.159 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:19:58.159 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:19:58.160 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:19:59.170 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-12-03T01:19:59.171 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:19:59.247 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2022-12-03T01:19:59.280 DEBUG:teuthology.parallel:result is None 2022-12-03T01:19:59.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-12-03T01:19:59.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:19:59.848 DEBUG:teuthology.orchestra.run.smithi092:> true 2022-12-03T01:20:00.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-12-03T01:20:00.388 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-56-generic"... 2022-12-03T01:20:00.388 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2022-12-03T01:20:00.441 INFO:teuthology.orchestra.run.smithi092.stdout:5.15.0-56-generic 2022-12-03T01:20:00.441 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:20:00.442 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:20:00.442 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:20:00.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-12-03T01:20:00.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:20:01.001 DEBUG:teuthology.orchestra.run.smithi138:> true 2022-12-03T01:20:01.443 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-12-03T01:20:01.444 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:20:01.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-12-03T01:20:01.450 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.15.0-56-generic"... 2022-12-03T01:20:01.450 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2022-12-03T01:20:01.499 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-56-generic 2022-12-03T01:20:01.500 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5.15.0-56-generic 2022-12-03T01:20:01.501 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-12-03T01:20:01.501 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-12-03T01:20:01.510 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2022-12-03T01:20:01.534 DEBUG:teuthology.parallel:result is None 2022-12-03T01:20:02.504 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-12-03T01:20:02.505 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-12-03T01:20:02.854 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2022-12-03T01:20:02.876 DEBUG:teuthology.parallel:result is None 2022-12-03T01:20:02.876 INFO:teuthology.run_tasks:Running task internal.base... 2022-12-03T01:20:02.898 INFO:teuthology.task.internal:Creating test directory... 2022-12-03T01:20:02.902 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:20:02.906 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:20:02.919 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-12-03T01:20:02.940 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-12-03T01:20:02.954 INFO:teuthology.run_tasks:Running task internal.archive... 2022-12-03T01:20:02.984 INFO:teuthology.task.internal:Creating archive directory... 2022-12-03T01:20:02.985 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:20:02.988 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:20:02.995 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-12-03T01:20:03.022 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-12-03T01:20:03.036 INFO:teuthology.task.internal:Enabling coredump saving... 2022-12-03T01:20:03.037 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:20:03.060 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:20:03.064 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-12-03T01:20:03.093 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.097 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.098 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.104 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.104 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.110 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-12-03T01:20:03.111 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-12-03T01:20:03.122 INFO:teuthology.task.internal:Configuring sudo... 2022-12-03T01:20:03.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:20:03.156 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:20:03.158 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-12-03T01:20:03.199 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-12-03T01:20:03.219 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-12-03T01:20:03.220 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:20:03.241 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:20:03.244 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-12-03T01:20:03.265 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:20:03.301 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:20:03.356 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:20:03.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:20:03.418 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:20:03.428 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:20:03.481 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:20:03.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:20:03.546 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-12-03T01:20:03.558 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-12-03T01:20:03.611 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:20:03.611 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-12-03T01:20:03.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2022-12-03T01:20:03.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2022-12-03T01:20:03.692 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2022-12-03T01:20:03.773 INFO:teuthology.run_tasks:Running task internal.timer... 2022-12-03T01:20:03.781 INFO:teuthology.task.internal:Starting timer... 2022-12-03T01:20:03.782 INFO:teuthology.run_tasks:Running task pcp... 2022-12-03T01:20:03.811 INFO:teuthology.run_tasks:Running task selinux... 2022-12-03T01:20:03.847 DEBUG:teuthology.task.selinux:Excluding smithi092: OS 'ubuntu' does not support SELinux 2022-12-03T01:20:03.847 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2022-12-03T01:20:03.848 DEBUG:teuthology.task.selinux:Excluding smithi138: OS 'ubuntu' does not support SELinux 2022-12-03T01:20:03.848 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-12-03T01:20:03.848 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-12-03T01:20:03.849 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-12-03T01:20:03.849 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-12-03T01:20:03.890 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2022-12-03T01:20:03.899 INFO:teuthology.repo_utils:Fetching main from origin 2022-12-03T01:20:04.108 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2022-12-03T01:20:04.141 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-12-03T01:20:04.144 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2022-12-03T01:25:10.225 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2022-12-03T01:25:10.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2022-12-03T01:25:10.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:10.376 DEBUG:teuthology.orchestra.run.smithi092:> true 2022-12-03T01:25:10.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-12-03T01:25:10.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:10.634 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-12-03T01:25:10.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-12-03T01:25:10.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-12-03T01:25:10.857 DEBUG:teuthology.orchestra.run.smithi138:> true 2022-12-03T01:25:10.978 INFO:teuthology.run_tasks:Running task clock... 2022-12-03T01:25:10.997 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-12-03T01:25:10.998 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:25:11.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:25:11.005 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:25:11.006 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:25:11.009 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-12-03T01:25:11.010 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:25:11.033 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:25:11.034 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Command line: ntpd -gq 2022-12-03T01:25:11.035 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: proto: precision = 0.070 usec (-24) 2022-12-03T01:25:11.037 INFO:teuthology.orchestra.run.smithi092.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.039 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.039 INFO:teuthology.orchestra.run.smithi092.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.041 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.041 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:25:11.041 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:25:11.042 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:25:11.042 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen normally on 3 enp3s0f1 172.21.15.92:123 2022-12-03T01:25:11.043 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen normally on 4 lo [::1]:123 2022-12-03T01:25:11.044 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cd45%5]:123 2022-12-03T01:25:11.044 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:11 ntpd[17143]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:25:11.045 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:25:11.046 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Command line: ntpd -gq 2022-12-03T01:25:11.046 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: proto: precision = 0.082 usec (-23) 2022-12-03T01:25:11.047 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.048 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.049 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.052 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.052 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:25:11.053 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:25:11.053 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:25:11.053 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2022-12-03T01:25:11.053 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen normally on 4 lo [::1]:123 2022-12-03T01:25:11.054 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2022-12-03T01:25:11.054 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:11 ntpd[17188]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:25:11.055 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-12-03T01:25:11.055 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Command line: ntpd -gq 2022-12-03T01:25:11.055 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: proto: precision = 0.070 usec (-24) 2022-12-03T01:25:11.056 INFO:teuthology.orchestra.run.smithi138.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.056 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-12-03T01:25:11.056 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.057 INFO:teuthology.orchestra.run.smithi138.stderr:restrict ::: KOD does nothing without LIMITED. 2022-12-03T01:25:11.058 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen and drop on 0 v6wildcard [::]:123 2022-12-03T01:25:11.058 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-12-03T01:25:11.058 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen normally on 2 lo 127.0.0.1:123 2022-12-03T01:25:11.059 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen normally on 3 enp3s0f1 172.21.15.138:123 2022-12-03T01:25:11.059 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen normally on 4 lo [::1]:123 2022-12-03T01:25:11.059 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f71%5]:123 2022-12-03T01:25:11.059 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:11 ntpd[17146]: Listening on routing socket on fd #22 for interface updates 2022-12-03T01:25:12.033 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.034 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.034 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:12.039 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:12.039 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.040 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.040 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:12.041 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:12 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:12.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:12.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:12.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:12.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:12 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:13.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:13.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:13.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:13.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:13.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:13.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:13 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:14.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:14.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:14.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:14.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:14 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:14.040 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:14.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:14 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:14.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:14.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:14.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:14.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:14.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:14.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:14 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:15.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:15.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:15.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:15.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:15.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:15.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:15.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:15.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:15.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:15.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:15.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:15.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:15.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:15.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:15 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.034 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.037 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.039 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.039 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:16 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.040 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:16 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.058 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.058 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.058 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.058 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.058 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.059 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.059 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.059 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.059 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:16.060 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:16.060 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:16.060 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:16.060 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:16 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:17.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:17.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:17.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:17.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:17.042 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:17.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:17.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:17.043 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:17.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:17.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:17.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:17.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:17 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:18.034 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.034 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.035 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:18.036 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:18.037 INFO:teuthology.orchestra.run.smithi092.stdout: 3 Dec 01:25:18 ntpd[17143]: ntpd: time slew -0.004026 s 2022-12-03T01:25:18.037 INFO:teuthology.orchestra.run.smithi092.stdout:ntpd: time slew -0.004026s 2022-12-03T01:25:18.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:25:18.038 INFO:teuthology.orchestra.run.smithi092.stderr: 3 Dec 01:25:18 ntpd[17143]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:25:18.040 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:18.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:18 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:18.053 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.054 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:18.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:18.055 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:18.056 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:18.056 INFO:teuthology.orchestra.run.smithi138.stdout: 3 Dec 01:25:18 ntpd[17146]: ntpd: time slew -0.003016 s 2022-12-03T01:25:18.056 INFO:teuthology.orchestra.run.smithi138.stdout:ntpd: time slew -0.003016s 2022-12-03T01:25:18.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:25:18.057 INFO:teuthology.orchestra.run.smithi138.stderr: 3 Dec 01:25:18 ntpd[17146]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:25:18.101 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:25:18.102 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2022-12-03T01:25:18.102 INFO:teuthology.orchestra.run.smithi092.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.103 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:25:18.140 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2022-12-03T01:25:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:19.040 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:19.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:19.041 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:19.044 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:19.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:19.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:19.045 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:19.047 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:19.048 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-12-03T01:25:19.048 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/rawstats.20221203: Permission denied 2022-12-03T01:25:19.049 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-12-03T01:25:19.049 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/peerstats.20221203: Permission denied 2022-12-03T01:25:19.050 INFO:teuthology.orchestra.run.smithi121.stdout: 3 Dec 01:25:19 ntpd[17188]: ntpd: time slew -0.000971 s 2022-12-03T01:25:19.051 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew -0.000971s 2022-12-03T01:25:19.052 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-12-03T01:25:19.052 INFO:teuthology.orchestra.run.smithi121.stderr: 3 Dec 01:25:19 ntpd[17188]: can't open /var/log/ntpstats/loopstats.20221203: Permission denied 2022-12-03T01:25:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:25:19.107 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-12-03T01:25:19.107 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:19.108 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-12-03T01:25:19.110 INFO:teuthology.run_tasks:Running task kubeadm... 2022-12-03T01:25:19.126 INFO:tasks.kubeadm:Kubeadm start 2022-12-03T01:25:19.127 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-12-03T01:25:19.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:25:19.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:25:19.155 INFO:teuthology.orchestra.run.smithi092.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:25:19.157 DEBUG:teuthology.orchestra.run.smithi138:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-12-03T01:25:19.164 INFO:teuthology.orchestra.run.smithi092.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:25:19.173 INFO:teuthology.orchestra.run.smithi092.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:25:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:25:19.184 INFO:teuthology.orchestra.run.smithi138.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-12-03T01:25:19.186 INFO:teuthology.orchestra.run.smithi121.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:25:19.192 INFO:teuthology.orchestra.run.smithi138.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-12-03T01:25:19.194 INFO:teuthology.orchestra.run.smithi121.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:25:19.201 INFO:teuthology.orchestra.run.smithi138.stdout:net.ipv4.ip_forward = 1 2022-12-03T01:25:19.261 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:25:19.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:25:19.295 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:25:19.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:25:19.358 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:25:19.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:25:19.389 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:25:19.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:25:19.449 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:25:19.450 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-12-03T01:25:19.467 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:25:19.468 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/docker/daemon.json 2022-12-03T01:25:19.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl restart docker || true 2022-12-03T01:25:19.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl restart docker || true 2022-12-03T01:25:19.534 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl restart docker || true 2022-12-03T01:25:20.655 INFO:tasks.kubeadm:Identifying pod subnet 2022-12-03T01:25:20.667 INFO:tasks.kubeadm:172.21.15.92 is in 172.21.0.0/20 at pos 3931 2022-12-03T01:25:20.740 INFO:tasks.kubeadm:Pod subnet: 10.250.216.0/21 2022-12-03T01:25:20.741 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-12-03T01:25:20.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:25:20.744 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:25:20.747 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-12-03T01:25:20.767 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:20.768 INFO:teuthology.orchestra.run.smithi121.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:20.768 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:20.769 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:20.769 INFO:teuthology.orchestra.run.smithi138.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:20.770 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:20.770 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:20.770 INFO:teuthology.orchestra.run.smithi092.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:20.771 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:20.898 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:20.899 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:20.903 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:20.904 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:20.904 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:20.911 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:20.920 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:20.931 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:21.024 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:21.024 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:21.092 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:21.172 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:22.455 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:25:22.464 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:25:22.598 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:25:22.638 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:25:22.639 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:25:22.644 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:25:22.645 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:25:22.744 INFO:teuthology.orchestra.run.smithi138.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:22.746 INFO:teuthology.orchestra.run.smithi092.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:22.757 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:22.759 INFO:teuthology.orchestra.run.smithi138.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:22.759 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:22.760 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:22.761 INFO:teuthology.orchestra.run.smithi092.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:22.761 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:22.780 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:25:22.781 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:25:22.810 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:25:22.815 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:25:22.876 INFO:teuthology.orchestra.run.smithi121.stdout:49 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:22.890 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:22.891 INFO:teuthology.orchestra.run.smithi121.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:22.891 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:22.944 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:25:22.977 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:25:22.977 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:25:23.000 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:25:23.001 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:25:23.112 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:25:23.112 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:25:23.157 INFO:teuthology.orchestra.run.smithi138.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:25:23.158 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2022-12-03T01:25:23.159 INFO:teuthology.orchestra.run.smithi138.stdout: apt-transport-https 2022-12-03T01:25:23.159 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages will be upgraded: 2022-12-03T01:25:23.160 INFO:teuthology.orchestra.run.smithi138.stdout: curl libcurl4 2022-12-03T01:25:23.195 INFO:teuthology.orchestra.run.smithi092.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:25:23.195 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-12-03T01:25:23.196 INFO:teuthology.orchestra.run.smithi092.stdout: apt-transport-https 2022-12-03T01:25:23.197 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2022-12-03T01:25:23.198 INFO:teuthology.orchestra.run.smithi092.stdout: curl libcurl4 2022-12-03T01:25:23.216 INFO:teuthology.orchestra.run.smithi138.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:23.216 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 398 kB of archives. 2022-12-03T01:25:23.216 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:25:23.217 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:25:23.236 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:25:23.294 INFO:teuthology.orchestra.run.smithi121.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-12-03T01:25:23.294 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-12-03T01:25:23.295 INFO:teuthology.orchestra.run.smithi121.stdout: apt-transport-https 2022-12-03T01:25:23.295 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-12-03T01:25:23.297 INFO:teuthology.orchestra.run.smithi121.stdout: curl libcurl4 2022-12-03T01:25:23.314 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:25:23.349 INFO:teuthology.orchestra.run.smithi121.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:23.350 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 398 kB of archives. 2022-12-03T01:25:23.350 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:25:23.351 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:25:23.366 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:25:23.377 INFO:teuthology.orchestra.run.smithi092.stdout:2 upgraded, 1 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:23.378 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 398 kB of archives. 2022-12-03T01:25:23.378 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 164 kB of additional disk space will be used. 2022-12-03T01:25:23.378 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-12-03T01:25:23.427 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:25:23.458 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 curl amd64 7.68.0-1ubuntu2.14 [161 kB] 2022-12-03T01:25:23.572 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:23.572 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:23.572 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:23.591 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:23.591 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:23.591 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:23.597 INFO:teuthology.orchestra.run.smithi138.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:23.648 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 398 kB in 0s (2,281 kB/s) 2022-12-03T01:25:23.689 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:23.690 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:23.690 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:23.713 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:23.719 INFO:teuthology.orchestra.run.smithi121.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:23.744 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:25:23.761 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu2.14 [235 kB] 2022-12-03T01:25:23.770 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 398 kB in 0s (2,714 kB/s) 2022-12-03T01:25:23.782 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:25:23.786 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:25:23.825 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:25:23.862 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:25:23.900 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:25:23.903 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:25:23.953 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:25:24.085 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:24.126 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:24.127 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:24.127 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:24.151 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:24.152 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:24.152 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:24.157 INFO:teuthology.orchestra.run.smithi092.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:24.207 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 398 kB in 1s (597 kB/s) 2022-12-03T01:25:24.212 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:24.220 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:24.297 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package apt-transport-https. 2022-12-03T01:25:24.335 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109733 files and directories currently installed.) 2022-12-03T01:25:24.338 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-12-03T01:25:24.380 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-12-03T01:25:24.389 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:24.646 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../curl_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:24.648 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:24.750 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking curl (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:24.826 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:24.891 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:25.086 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:25.159 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:25:25.245 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../libcurl4_7.68.0-1ubuntu2.14_amd64.deb ... 2022-12-03T01:25:25.283 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:25.417 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:25.419 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:25:25.456 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcurl4:amd64 (7.68.0-1ubuntu2.14) over (7.68.0-1ubuntu2.13) ... 2022-12-03T01:25:25.516 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:25.535 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:25.617 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:25.760 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:25.812 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up apt-transport-https (2.0.9) ... 2022-12-03T01:25:25.919 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcurl4:amd64 (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:25.971 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:25:26.070 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up curl (7.68.0-1ubuntu2.14) ... 2022-12-03T01:25:26.204 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:25:26.205 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:26.522 INFO:teuthology.orchestra.run.smithi138.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:25:26.536 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:26.536 INFO:teuthology.orchestra.run.smithi138.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:26.536 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:26.632 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-12-03T01:25:26.667 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:26.677 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:26.697 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:26.780 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:26.787 INFO:teuthology.orchestra.run.smithi121.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:25:26.802 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:26.802 INFO:teuthology.orchestra.run.smithi121.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:26.803 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:26.925 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:26.930 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:26.944 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:25:26.947 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:27.040 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:27.214 INFO:teuthology.orchestra.run.smithi092.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-12-03T01:25:27.228 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:27.228 INFO:teuthology.orchestra.run.smithi092.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:27.229 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:27.233 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:25:27.310 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:25:27.476 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 70.8 kB in 1s (89.2 kB/s) 2022-12-03T01:25:27.478 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-12-03T01:25:27.479 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-12-03T01:25:27.548 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-12-03T01:25:27.560 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:25:27.628 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-12-03T01:25:27.651 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-12-03T01:25:27.735 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 70.8 kB in 1s (90.9 kB/s) 2022-12-03T01:25:28.124 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [61.4 kB] 2022-12-03T01:25:28.299 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 70.8 kB in 1s (76.6 kB/s) 2022-12-03T01:25:28.408 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:25:28.591 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:25:28.592 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:25:28.681 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:25:28.682 INFO:teuthology.orchestra.run.smithi138.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:28.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:28.696 INFO:teuthology.orchestra.run.smithi138.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:28.696 INFO:teuthology.orchestra.run.smithi138.stderr: 2022-12-03T01:25:28.751 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2022-12-03T01:25:28.863 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:25:28.863 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:25:28.937 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2022-12-03T01:25:28.939 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2022-12-03T01:25:28.960 INFO:teuthology.orchestra.run.smithi121.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:28.972 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:28.972 INFO:teuthology.orchestra.run.smithi121.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:28.973 INFO:teuthology.orchestra.run.smithi121.stderr: 2022-12-03T01:25:29.028 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-12-03T01:25:29.105 INFO:teuthology.orchestra.run.smithi138.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:25:29.106 INFO:teuthology.orchestra.run.smithi138.stdout:bridge-utils set to manually installed. 2022-12-03T01:25:29.106 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2022-12-03T01:25:29.106 INFO:teuthology.orchestra.run.smithi138.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:25:29.107 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2022-12-03T01:25:29.107 INFO:teuthology.orchestra.run.smithi138.stdout: nftables 2022-12-03T01:25:29.156 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2022-12-03T01:25:29.158 INFO:teuthology.orchestra.run.smithi138.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:25:29.212 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-12-03T01:25:29.213 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-12-03T01:25:29.226 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:29.227 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:25:29.227 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:25:29.227 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:25:29.233 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:25:29.282 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:25:29.315 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:25:29.364 INFO:teuthology.orchestra.run.smithi121.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:25:29.365 INFO:teuthology.orchestra.run.smithi121.stdout:bridge-utils set to manually installed. 2022-12-03T01:25:29.365 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-12-03T01:25:29.366 INFO:teuthology.orchestra.run.smithi121.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:25:29.366 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-12-03T01:25:29.367 INFO:teuthology.orchestra.run.smithi121.stdout: nftables 2022-12-03T01:25:29.417 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-12-03T01:25:29.418 INFO:teuthology.orchestra.run.smithi121.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:25:29.421 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:25:29.422 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:25:29.449 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:25:29.481 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:29.481 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:25:29.481 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:25:29.481 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:25:29.516 INFO:teuthology.orchestra.run.smithi092.stdout:47 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-12-03T01:25:29.525 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:29.526 INFO:teuthology.orchestra.run.smithi092.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-12-03T01:25:29.526 INFO:teuthology.orchestra.run.smithi092.stderr: 2022-12-03T01:25:29.528 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:25:29.561 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:25:29.583 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2022-12-03T01:25:29.702 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:25:29.748 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2022-12-03T01:25:29.748 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2022-12-03T01:25:29.895 INFO:teuthology.orchestra.run.smithi092.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-12-03T01:25:29.895 INFO:teuthology.orchestra.run.smithi092.stdout:bridge-utils set to manually installed. 2022-12-03T01:25:29.896 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2022-12-03T01:25:29.897 INFO:teuthology.orchestra.run.smithi092.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-12-03T01:25:29.897 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2022-12-03T01:25:29.897 INFO:teuthology.orchestra.run.smithi092.stdout: nftables 2022-12-03T01:25:29.929 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:25:29.947 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2022-12-03T01:25:29.948 INFO:teuthology.orchestra.run.smithi092.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-12-03T01:25:30.138 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 8 newly installed, 0 to remove and 47 not upgraded. 2022-12-03T01:25:30.138 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 81.6 MB of archives. 2022-12-03T01:25:30.139 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 327 MB of additional disk space will be used. 2022-12-03T01:25:30.139 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-12-03T01:25:30.144 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:25:30.230 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.25.0-00 [17.9 MB] 2022-12-03T01:25:30.337 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-12-03T01:25:30.488 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-12-03T01:25:30.510 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:25:30.705 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.1.1-00 [25.0 MB] 2022-12-03T01:25:30.726 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:25:30.914 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:25:31.011 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:25:31.098 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.25.4-00 [19.5 MB] 2022-12-03T01:25:31.241 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:25:31.344 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.25.4-00 [9,498 kB] 2022-12-03T01:25:31.464 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:25:31.498 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:31.499 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:31.507 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:31.508 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:31.508 INFO:teuthology.orchestra.run.smithi138.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:31.512 INFO:teuthology.orchestra.run.smithi138.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:31.531 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.25.4-00 [9,219 kB] 2022-12-03T01:25:31.563 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 81.6 MB in 2s (38.7 MB/s) 2022-12-03T01:25:31.643 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:25:31.689 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:25:31.692 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:25:31.725 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:25:31.853 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:31.854 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:31.854 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:31.861 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: unable to initialize frontend: Dialog 2022-12-03T01:25:31.861 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-12-03T01:25:31.861 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: falling back to frontend: Readline 2022-12-03T01:25:31.862 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:31.862 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:31.862 INFO:teuthology.orchestra.run.smithi121.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:31.868 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: unable to initialize frontend: Readline 2022-12-03T01:25:31.868 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: (This frontend requires a controlling tty.) 2022-12-03T01:25:31.868 INFO:teuthology.orchestra.run.smithi092.stderr:debconf: falling back to frontend: Teletype 2022-12-03T01:25:31.869 INFO:teuthology.orchestra.run.smithi121.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:31.873 INFO:teuthology.orchestra.run.smithi092.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-12-03T01:25:31.920 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 81.6 MB in 2s (37.0 MB/s) 2022-12-03T01:25:31.932 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 81.6 MB in 2s (48.6 MB/s) 2022-12-03T01:25:32.000 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:25:32.022 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package conntrack. 2022-12-03T01:25:32.023 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:25:32.033 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:25:32.047 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:25:32.050 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:25:32.064 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109737 files and directories currently installed.) 2022-12-03T01:25:32.067 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-12-03T01:25:32.078 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:25:32.090 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:25:32.110 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-12-03T01:25:32.418 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:25:32.421 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package cri-tools. 2022-12-03T01:25:32.431 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:25:32.434 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../1-cri-tools_1.25.0-00_amd64.deb ... 2022-12-03T01:25:32.471 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:25:32.476 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cri-tools (1.25.0-00) ... 2022-12-03T01:25:33.606 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:25:33.620 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:25:33.671 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:25:33.849 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:25:33.862 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:25:33.917 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:25:33.932 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:25:33.947 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:25:33.975 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:34.004 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ebtables. 2022-12-03T01:25:34.019 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-12-03T01:25:34.071 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-12-03T01:25:34.177 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:25:34.191 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:25:34.235 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:34.332 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package kubernetes-cni. 2022-12-03T01:25:34.346 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../3-kubernetes-cni_1.1.1-00_amd64.deb ... 2022-12-03T01:25:34.390 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:35.395 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package socat. 2022-12-03T01:25:35.409 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:25:35.444 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:25:35.671 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package socat. 2022-12-03T01:25:35.684 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:25:35.721 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:25:35.831 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:25:35.845 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:25:35.867 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2022-12-03T01:25:35.882 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-12-03T01:25:35.889 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:25:35.917 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.3.3-2) ... 2022-12-03T01:25:36.166 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:25:36.181 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:25:36.257 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:25:36.296 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package kubelet. 2022-12-03T01:25:36.309 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../5-kubelet_1.25.4-00_amd64.deb ... 2022-12-03T01:25:36.346 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking kubelet (1.25.4-00) ... 2022-12-03T01:25:38.276 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:25:38.289 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:25:38.334 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:25:38.543 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:25:38.556 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:25:38.593 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:25:38.756 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package kubectl. 2022-12-03T01:25:38.764 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../6-kubectl_1.25.4-00_amd64.deb ... 2022-12-03T01:25:38.806 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking kubectl (1.25.4-00) ... 2022-12-03T01:25:39.545 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:25:39.560 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:25:39.603 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:25:39.820 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:25:39.834 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:25:39.879 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:25:40.042 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package kubeadm. 2022-12-03T01:25:40.050 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../7-kubeadm_1.25.4-00_amd64.deb ... 2022-12-03T01:25:40.117 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking kubeadm (1.25.4-00) ... 2022-12-03T01:25:40.764 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:25:40.907 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:25:41.033 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:25:41.092 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:25:41.224 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:25:41.274 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-12-03T01:25:41.342 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:25:41.395 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up kubectl (1.25.4-00) ... 2022-12-03T01:25:41.521 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-12-03T01:25:41.728 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:25:41.854 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:25:41.980 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:42.061 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:25:42.106 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:25:42.187 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:25:42.232 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.3.3-2) ... 2022-12-03T01:25:42.325 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:25:42.338 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:42.358 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cri-tools (1.25.0-00) ... 2022-12-03T01:25:42.439 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:25:42.484 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up kubernetes-cni (1.1.1-00) ... 2022-12-03T01:25:42.626 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:25:42.627 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up kubelet (1.25.4-00) ... 2022-12-03T01:25:42.782 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:25:42.835 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-12-03T01:25:43.140 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:25:43.160 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:43.303 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up kubeadm (1.25.4-00) ... 2022-12-03T01:25:43.526 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:43.689 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-12-03T01:25:44.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:25:44.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:25:44.591 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-12-03T01:25:58.220 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:25:58.561 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:25:58.675 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.4 2022-12-03T01:26:01.859 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:26:01.935 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:26:02.127 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.4 2022-12-03T01:26:04.590 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:26:04.701 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:26:04.775 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.4 2022-12-03T01:26:07.515 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:26:07.933 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:26:08.275 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/kube-proxy:v1.25.4 2022-12-03T01:26:09.596 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:26:10.075 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:26:10.339 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/pause:3.8 2022-12-03T01:26:15.821 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:26:16.490 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:26:16.514 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/etcd:3.5.5-0 2022-12-03T01:26:19.044 INFO:teuthology.orchestra.run.smithi138.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:26:19.640 INFO:teuthology.orchestra.run.smithi092.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:26:19.922 INFO:teuthology.orchestra.run.smithi121.stdout:[config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3 2022-12-03T01:26:19.926 INFO:tasks.kubeadm:Token: abcdef.1jdeyxkj89w8w7dm 2022-12-03T01:26:19.926 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): '172.21.15.92', Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): '172.21.15.121', Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): '172.21.15.138'} 2022-12-03T01:26:19.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo kubeadm init --node-name smithi092 --token abcdef.1jdeyxkj89w8w7dm --pod-network-cidr 10.250.216.0/21 2022-12-03T01:26:20.192 INFO:teuthology.orchestra.run.smithi092.stdout:[init] Using Kubernetes version: v1.25.4 2022-12-03T01:26:20.193 INFO:teuthology.orchestra.run.smithi092.stdout:[preflight] Running pre-flight checks 2022-12-03T01:26:20.391 INFO:teuthology.orchestra.run.smithi092.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-12-03T01:26:20.391 INFO:teuthology.orchestra.run.smithi092.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-12-03T01:26:20.391 INFO:teuthology.orchestra.run.smithi092.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-12-03T01:26:20.579 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-12-03T01:26:20.781 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "ca" certificate and key 2022-12-03T01:26:21.004 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "apiserver" certificate and key 2022-12-03T01:26:21.004 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi092] and IPs [10.96.0.1 172.21.15.92] 2022-12-03T01:26:21.207 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-12-03T01:26:21.371 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-12-03T01:26:21.535 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-12-03T01:26:21.658 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "etcd/ca" certificate and key 2022-12-03T01:26:21.741 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "etcd/server" certificate and key 2022-12-03T01:26:21.741 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi092] and IPs [172.21.15.92 127.0.0.1 ::1] 2022-12-03T01:26:21.819 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "etcd/peer" certificate and key 2022-12-03T01:26:21.819 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi092] and IPs [172.21.15.92 127.0.0.1 ::1] 2022-12-03T01:26:21.903 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-12-03T01:26:22.137 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-12-03T01:26:22.276 INFO:teuthology.orchestra.run.smithi092.stdout:[certs] Generating "sa" key and public key 2022-12-03T01:26:22.276 INFO:teuthology.orchestra.run.smithi092.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-12-03T01:26:22.336 INFO:teuthology.orchestra.run.smithi092.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-12-03T01:26:22.618 INFO:teuthology.orchestra.run.smithi092.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-12-03T01:26:22.705 INFO:teuthology.orchestra.run.smithi092.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-12-03T01:26:22.873 INFO:teuthology.orchestra.run.smithi092.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-12-03T01:26:22.887 INFO:teuthology.orchestra.run.smithi092.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:26:22.915 INFO:teuthology.orchestra.run.smithi092.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:26:22.915 INFO:teuthology.orchestra.run.smithi092.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:26:23.104 INFO:teuthology.orchestra.run.smithi092.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-12-03T01:26:23.105 INFO:teuthology.orchestra.run.smithi092.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-12-03T01:26:23.107 INFO:teuthology.orchestra.run.smithi092.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-12-03T01:26:23.108 INFO:teuthology.orchestra.run.smithi092.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-12-03T01:26:23.109 INFO:teuthology.orchestra.run.smithi092.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-12-03T01:26:23.111 INFO:teuthology.orchestra.run.smithi092.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-12-03T01:26:44.642 INFO:teuthology.orchestra.run.smithi092.stdout:[apiclient] All control plane components are healthy after 21.530934 seconds 2022-12-03T01:26:44.642 INFO:teuthology.orchestra.run.smithi092.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-12-03T01:26:44.688 INFO:teuthology.orchestra.run.smithi092.stdout:[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster 2022-12-03T01:26:45.255 INFO:teuthology.orchestra.run.smithi092.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-12-03T01:26:45.256 INFO:teuthology.orchestra.run.smithi092.stdout:[mark-control-plane] Marking the node smithi092 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-12-03T01:26:45.256 INFO:teuthology.orchestra.run.smithi092.stdout:[mark-control-plane] Marking the node smithi092 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] 2022-12-03T01:26:45.805 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Using token: abcdef.1jdeyxkj89w8w7dm 2022-12-03T01:26:45.806 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-12-03T01:26:45.905 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-12-03T01:26:45.930 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-12-03T01:26:45.938 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-12-03T01:26:45.963 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-12-03T01:26:46.008 INFO:teuthology.orchestra.run.smithi092.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-12-03T01:26:46.058 INFO:teuthology.orchestra.run.smithi092.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-12-03T01:26:46.655 INFO:teuthology.orchestra.run.smithi092.stdout:[addons] Applied essential addon: CoreDNS 2022-12-03T01:26:46.805 INFO:teuthology.orchestra.run.smithi092.stdout:[addons] Applied essential addon: kube-proxy 2022-12-03T01:26:46.807 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.807 INFO:teuthology.orchestra.run.smithi092.stdout:Your Kubernetes control-plane has initialized successfully! 2022-12-03T01:26:46.808 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.808 INFO:teuthology.orchestra.run.smithi092.stdout:To start using your cluster, you need to run the following as a regular user: 2022-12-03T01:26:46.808 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.808 INFO:teuthology.orchestra.run.smithi092.stdout: mkdir -p $HOME/.kube 2022-12-03T01:26:46.809 INFO:teuthology.orchestra.run.smithi092.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-12-03T01:26:46.809 INFO:teuthology.orchestra.run.smithi092.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-12-03T01:26:46.809 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.809 INFO:teuthology.orchestra.run.smithi092.stdout:Alternatively, if you are the root user, you can run: 2022-12-03T01:26:46.810 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.810 INFO:teuthology.orchestra.run.smithi092.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-12-03T01:26:46.810 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.810 INFO:teuthology.orchestra.run.smithi092.stdout:You should now deploy a pod network to the cluster. 2022-12-03T01:26:46.810 INFO:teuthology.orchestra.run.smithi092.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-12-03T01:26:46.811 INFO:teuthology.orchestra.run.smithi092.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-12-03T01:26:46.811 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.811 INFO:teuthology.orchestra.run.smithi092.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-12-03T01:26:46.811 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:26:46.812 INFO:teuthology.orchestra.run.smithi092.stdout:kubeadm join 172.21.15.92:6443 --token abcdef.1jdeyxkj89w8w7dm \ 2022-12-03T01:26:46.812 INFO:teuthology.orchestra.run.smithi092.stdout: --discovery-token-ca-cert-hash sha256:39a6e58b4e30fa41844aae762da174c683bae48eec68c539034447962e3a817e 2022-12-03T01:26:46.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo kubeadm join 172.21.15.92:6443 --node-name smithi121 --token abcdef.1jdeyxkj89w8w7dm --discovery-token-unsafe-skip-ca-verification 2022-12-03T01:26:46.817 DEBUG:teuthology.orchestra.run.smithi138:> sudo kubeadm join 172.21.15.92:6443 --node-name smithi138 --token abcdef.1jdeyxkj89w8w7dm --discovery-token-unsafe-skip-ca-verification 2022-12-03T01:26:46.856 INFO:teuthology.orchestra.run.smithi138.stdout:[preflight] Running pre-flight checks 2022-12-03T01:26:46.857 INFO:teuthology.orchestra.run.smithi121.stdout:[preflight] Running pre-flight checks 2022-12-03T01:26:59.666 INFO:teuthology.orchestra.run.smithi121.stdout:[preflight] Reading configuration from the cluster... 2022-12-03T01:26:59.667 INFO:teuthology.orchestra.run.smithi121.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:26:59.742 INFO:teuthology.orchestra.run.smithi121.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:26:59.770 INFO:teuthology.orchestra.run.smithi121.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:26:59.770 INFO:teuthology.orchestra.run.smithi121.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:26:59.897 INFO:teuthology.orchestra.run.smithi138.stdout:[preflight] Reading configuration from the cluster... 2022-12-03T01:26:59.897 INFO:teuthology.orchestra.run.smithi138.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:26:59.942 INFO:teuthology.orchestra.run.smithi138.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-12-03T01:26:59.983 INFO:teuthology.orchestra.run.smithi121.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-12-03T01:27:00.026 INFO:teuthology.orchestra.run.smithi138.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-12-03T01:27:00.026 INFO:teuthology.orchestra.run.smithi138.stdout:[kubelet-start] Starting the kubelet 2022-12-03T01:27:00.245 INFO:teuthology.orchestra.run.smithi138.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-12-03T01:27:13.528 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:27:13.528 INFO:teuthology.orchestra.run.smithi121.stdout:This node has joined the cluster: 2022-12-03T01:27:13.528 INFO:teuthology.orchestra.run.smithi121.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-12-03T01:27:13.528 INFO:teuthology.orchestra.run.smithi121.stdout:* The Kubelet was informed of the new secure connection details. 2022-12-03T01:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-12-03T01:27:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:27:14.304 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:27:14.304 INFO:teuthology.orchestra.run.smithi138.stdout:This node has joined the cluster: 2022-12-03T01:27:14.304 INFO:teuthology.orchestra.run.smithi138.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-12-03T01:27:14.304 INFO:teuthology.orchestra.run.smithi138.stdout:* The Kubelet was informed of the new secure connection details. 2022-12-03T01:27:14.305 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:27:14.305 INFO:teuthology.orchestra.run.smithi138.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-12-03T01:27:14.305 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:27:14.315 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:14.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-12-03T01:27:14.334 INFO:tasks.kubeadm:Setting up kubectl 2022-12-03T01:27:14.334 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:27:14.394 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:27:14.411 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-12-03T01:27:14.431 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:14.432 DEBUG:teuthology.orchestra.run.smithi092:> dd of=.kube/config 2022-12-03T01:27:14.440 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:14.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/root/.kube/config 2022-12-03T01:27:14.502 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:14.503 DEBUG:teuthology.orchestra.run.smithi121:> dd of=.kube/config 2022-12-03T01:27:14.510 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:14.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/root/.kube/config 2022-12-03T01:27:14.568 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:14.569 DEBUG:teuthology.orchestra.run.smithi138:> dd of=.kube/config 2022-12-03T01:27:14.577 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:14.578 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/root/.kube/config 2022-12-03T01:27:14.634 DEBUG:teuthology.orchestra.run.smithi092:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-12-03T01:27:14.650 INFO:teuthology.orchestra.run.smithi092.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-12-03T01:27:14.650 INFO:teuthology.orchestra.run.smithi092.stderr: Dload Upload Total Spent Left Speed 2022-12-03T01:27:14.809 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.809 INFO:teuthology.orchestra.run.smithi092.stdout:kind: Namespace 2022-12-03T01:27:14.810 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: v1 2022-12-03T01:27:14.810 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.810 INFO:teuthology.orchestra.run.smithi092.stdout: name: kube-flannel 2022-12-03T01:27:14.811 INFO:teuthology.orchestra.run.smithi092.stdout: labels: 2022-12-03T01:27:14.811 INFO:teuthology.orchestra.run.smithi092.stdout: pod-security.kubernetes.io/enforce: privileged 2022-12-03T01:27:14.811 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.812 INFO:teuthology.orchestra.run.smithi092.stdout:kind: ClusterRole 2022-12-03T01:27:14.812 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-12-03T01:27:14.812 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.812 INFO:teuthology.orchestra.run.smithi092.stdout: name: flannel 2022-12-03T01:27:14.813 INFO:teuthology.orchestra.run.smithi092.stdout:rules: 2022-12-03T01:27:14.813 INFO:teuthology.orchestra.run.smithi092.stdout:- apiGroups: 2022-12-03T01:27:14.813 INFO:teuthology.orchestra.run.smithi092.stdout: - "" 2022-12-03T01:27:14.814 INFO:teuthology.orchestra.run.smithi092.stdout: resources: 2022-12-03T01:27:14.814 INFO:teuthology.orchestra.run.smithi092.stdout: - pods 2022-12-03T01:27:14.815 INFO:teuthology.orchestra.run.smithi092.stdout: verbs: 2022-12-03T01:27:14.815 INFO:teuthology.orchestra.run.smithi092.stdout: - get 2022-12-03T01:27:14.816 INFO:teuthology.orchestra.run.smithi092.stdout:- apiGroups: 2022-12-03T01:27:14.816 INFO:teuthology.orchestra.run.smithi092.stdout: - "" 2022-12-03T01:27:14.817 INFO:teuthology.orchestra.run.smithi092.stdout: resources: 2022-12-03T01:27:14.817 INFO:teuthology.orchestra.run.smithi092.stdout: - nodes 2022-12-03T01:27:14.818 INFO:teuthology.orchestra.run.smithi092.stdout: verbs: 2022-12-03T01:27:14.818 INFO:teuthology.orchestra.run.smithi092.stdout: - get 2022-12-03T01:27:14.819 INFO:teuthology.orchestra.run.smithi092.stdout: - list 2022-12-03T01:27:14.819 INFO:teuthology.orchestra.run.smithi092.stdout: - watch 2022-12-03T01:27:14.820 INFO:teuthology.orchestra.run.smithi092.stdout:- apiGroups: 2022-12-03T01:27:14.820 INFO:teuthology.orchestra.run.smithi092.stdout: - "" 2022-12-03T01:27:14.820 INFO:teuthology.orchestra.run.smithi092.stdout: resources: 2022-12-03T01:27:14.821 INFO:teuthology.orchestra.run.smithi092.stdout: - nodes/status 2022-12-03T01:27:14.821 INFO:teuthology.orchestra.run.smithi092.stdout: verbs: 2022-12-03T01:27:14.822 INFO:teuthology.orchestra.run.smithi092.stdout: - patch 2022-12-03T01:27:14.822 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.823 INFO:teuthology.orchestra.run.smithi092.stdout:kind: ClusterRoleBinding 2022-12-03T01:27:14.823 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-12-03T01:27:14.824 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.824 INFO:teuthology.orchestra.run.smithi092.stdout: name: flannel 2022-12-03T01:27:14.824 INFO:teuthology.orchestra.run.smithi092.stdout:roleRef: 2022-12-03T01:27:14.825 INFO:teuthology.orchestra.run.smithi092.stdout: apiGroup: rbac.authorization.k8s.io 2022-12-03T01:27:14.825 INFO:teuthology.orchestra.run.smithi092.stdout: kind: ClusterRole 2022-12-03T01:27:14.826 INFO:teuthology.orchestra.run.smithi092.stdout: name: flannel 2022-12-03T01:27:14.826 INFO:teuthology.orchestra.run.smithi092.stdout:subjects: 2022-12-03T01:27:14.826 INFO:teuthology.orchestra.run.smithi092.stdout:- kind: ServiceAccount 2022-12-03T01:27:14.827 INFO:teuthology.orchestra.run.smithi092.stdout: name: flannel 2022-12-03T01:27:14.827 INFO:teuthology.orchestra.run.smithi092.stdout: namespace: kube-flannel 2022-12-03T01:27:14.828 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.828 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: v1 2022-12-03T01:27:14.828 INFO:teuthology.orchestra.run.smithi092.stdout:kind: ServiceAccount 2022-12-03T01:27:14.828 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.829 INFO:teuthology.orchestra.run.smithi092.stdout: name: flannel 2022-12-03T01:27:14.829 INFO:teuthology.orchestra.run.smithi092.stdout: namespace: kube-flannel 2022-12-03T01:27:14.829 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.829 INFO:teuthology.orchestra.run.smithi092.stdout:kind: ConfigMap 2022-12-03T01:27:14.830 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: v1 2022-12-03T01:27:14.830 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.830 INFO:teuthology.orchestra.run.smithi092.stdout: name: kube-flannel-cfg 2022-12-03T01:27:14.830 INFO:teuthology.orchestra.run.smithi092.stdout: namespace: kube-flannel 2022-12-03T01:27:14.831 INFO:teuthology.orchestra.run.smithi092.stdout: labels: 2022-12-03T01:27:14.831 INFO:teuthology.orchestra.run.smithi092.stdout: tier: node 2022-12-03T01:27:14.831 INFO:teuthology.orchestra.run.smithi092.stdout: app: flannel 2022-12-03T01:27:14.832 INFO:teuthology.orchestra.run.smithi092.stdout:data: 2022-12-03T01:27:14.832 INFO:teuthology.orchestra.run.smithi092.stdout: cni-conf.json: | 2022-12-03T01:27:14.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2022-12-03T01:27:14.832 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "cbr0", 2022-12-03T01:27:14.833 INFO:teuthology.orchestra.run.smithi092.stdout: "cniVersion": "0.3.1", 2022-12-03T01:27:14.833 INFO:teuthology.orchestra.run.smithi092.stdout: "plugins": [ 2022-12-03T01:27:14.833 INFO:teuthology.orchestra.run.smithi092.stdout: { 2022-12-03T01:27:14.833 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "flannel", 2022-12-03T01:27:14.834 INFO:teuthology.orchestra.run.smithi092.stdout: "delegate": { 2022-12-03T01:27:14.834 INFO:teuthology.orchestra.run.smithi092.stdout: "hairpinMode": true, 2022-12-03T01:27:14.834 INFO:teuthology.orchestra.run.smithi092.stdout: "isDefaultGateway": true 2022-12-03T01:27:14.834 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.834 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2022-12-03T01:27:14.835 INFO:teuthology.orchestra.run.smithi092.stdout: { 2022-12-03T01:27:14.835 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "portmap", 2022-12-03T01:27:14.835 INFO:teuthology.orchestra.run.smithi092.stdout: "capabilities": { 2022-12-03T01:27:14.835 INFO:teuthology.orchestra.run.smithi092.stdout: "portMappings": true 2022-12-03T01:27:14.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.836 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2022-12-03T01:27:14.836 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.837 INFO:teuthology.orchestra.run.smithi092.stdout: net-conf.json: | 2022-12-03T01:27:14.837 INFO:teuthology.orchestra.run.smithi092.stdout: { 2022-12-03T01:27:14.837 INFO:teuthology.orchestra.run.smithi092.stdout: "Network": "10.244.0.0/16", 2022-12-03T01:27:14.837 INFO:teuthology.orchestra.run.smithi092.stdout: "Backend": { 2022-12-03T01:27:14.838 INFO:teuthology.orchestra.run.smithi092.stdout: "Type": "vxlan" 2022-12-03T01:27:14.838 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.838 INFO:teuthology.orchestra.run.smithi092.stdout: } 2022-12-03T01:27:14.838 INFO:teuthology.orchestra.run.smithi092.stdout:--- 2022-12-03T01:27:14.838 INFO:teuthology.orchestra.run.smithi092.stdout:apiVersion: apps/v1 2022-12-03T01:27:14.839 INFO:teuthology.orchestra.run.smithi092.stdout:kind: DaemonSet 2022-12-03T01:27:14.839 INFO:teuthology.orchestra.run.smithi092.stdout:metadata: 2022-12-03T01:27:14.839 INFO:teuthology.orchestra.run.smithi092.stdout: name: kube-flannel-ds 2022-12-03T01:27:14.839 INFO:teuthology.orchestra.run.smithi092.stdout: namespace: kube-flannel 2022-12-03T01:27:14.840 INFO:teuthology.orchestra.run.smithi092.stdout: labels: 2022-12-03T01:27:14.840 INFO:teuthology.orchestra.run.smithi092.stdout: tier: node 2022-12-03T01:27:14.840 INFO:teuthology.orchestra.run.smithi092.stdout: app: flannel 2022-12-03T01:27:14.840 INFO:teuthology.orchestra.run.smithi092.stdout:spec: 2022-12-03T01:27:14.840 INFO:teuthology.orchestra.run.smithi092.stdout: selector: 2022-12-03T01:27:14.841 INFO:teuthology.orchestra.run.smithi092.stdout: matchLabels: 2022-12-03T01:27:14.841 INFO:teuthology.orchestra.run.smithi092.stdout: app: flannel 2022-12-03T01:27:14.841 INFO:teuthology.orchestra.run.smithi092.stdout: template: 2022-12-03T01:27:14.841 INFO:teuthology.orchestra.run.smithi092.stdout: metadata: 2022-12-03T01:27:14.842 INFO:teuthology.orchestra.run.smithi092.stdout: labels: 2022-12-03T01:27:14.842 INFO:teuthology.orchestra.run.smithi092.stdout: tier: node 2022-12-03T01:27:14.842 INFO:teuthology.orchestra.run.smithi092.stdout: app: flannel 2022-12-03T01:27:14.842 INFO:teuthology.orchestra.run.smithi092.stdout: spec: 2022-12-03T01:27:14.843 INFO:teuthology.orchestra.run.smithi092.stdout: affinity: 2022-12-03T01:27:14.843 INFO:teuthology.orchestra.run.smithi092.stdout: nodeAffinity: 2022-12-03T01:27:14.843 INFO:teuthology.orchestra.run.smithi092.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-12-03T01:27:14.843 INFO:teuthology.orchestra.run.smithi092.stdout: nodeSelectorTerms: 2022-12-03T01:27:14.844 INFO:teuthology.orchestra.run.smithi092.stdout: - matchExpressions: 2022-12-03T01:27:14.844 INFO:teuthology.orchestra.run.smithi092.stdout: - key: kubernetes.io/os 2022-12-03T01:27:14.844 INFO:teuthology.orchestra.run.smithi092.stdout: operator: In 2022-12-03T01:27:14.844 INFO:teuthology.orchestra.run.smithi092.stdout: values: 2022-12-03T01:27:14.844 INFO:teuthology.orchestra.run.smithi092.stdout: - linux 2022-12-03T01:27:14.845 INFO:teuthology.orchestra.run.smithi092.stdout: hostNetwork: true 2022-12-03T01:27:14.845 INFO:teuthology.orchestra.run.smithi092.stdout: priorityClassName: system-node-critical 2022-12-03T01:27:14.845 INFO:teuthology.orchestra.run.smithi092.stdout: tolerations: 2022-12-03T01:27:14.845 INFO:teuthology.orchestra.run.smithi092.stdout: - operator: Exists 2022-12-03T01:27:14.846 INFO:teuthology.orchestra.run.smithi092.stdout: effect: NoSchedule 2022-12-03T01:27:14.846 INFO:teuthology.orchestra.run.smithi092.stdout: serviceAccountName: flannel 2022-12-03T01:27:14.846 INFO:teuthology.orchestra.run.smithi092.stdout: initContainers: 2022-12-03T01:27:14.846 INFO:teuthology.orchestra.run.smithi092.stdout: - name: install-cni-plugin 2022-12-03T01:27:14.846 INFO:teuthology.orchestra.run.smithi092.stdout: #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-12-03T01:27:14.847 INFO:teuthology.orchestra.run.smithi092.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 2022-12-03T01:27:14.847 INFO:teuthology.orchestra.run.smithi092.stdout: command: 2022-12-03T01:27:14.847 INFO:teuthology.orchestra.run.smithi092.stdout: - cp 2022-12-03T01:27:14.847 INFO:teuthology.orchestra.run.smithi092.stdout: args: 2022-12-03T01:27:14.848 INFO:teuthology.orchestra.run.smithi092.stdout: - -f 2022-12-03T01:27:14.848 INFO:teuthology.orchestra.run.smithi092.stdout: - /flannel 2022-12-03T01:27:14.848 INFO:teuthology.orchestra.run.smithi092.stdout: - /opt/cni/bin/flannel 2022-12-03T01:27:14.848 INFO:teuthology.orchestra.run.smithi092.stdout: volumeMounts: 2022-12-03T01:27:14.848 INFO:teuthology.orchestra.run.smithi092.stdout: - name: cni-plugin 2022-12-03T01:27:14.849 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /opt/cni/bin 2022-12-03T01:27:14.849 INFO:teuthology.orchestra.run.smithi092.stdout: - name: install-cni 2022-12-03T01:27:14.849 INFO:teuthology.orchestra.run.smithi092.stdout: #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply) 2022-12-03T01:27:14.849 INFO:teuthology.orchestra.run.smithi092.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2 2022-12-03T01:27:14.850 INFO:teuthology.orchestra.run.smithi092.stdout: command: 2022-12-03T01:27:14.850 INFO:teuthology.orchestra.run.smithi092.stdout: - cp 2022-12-03T01:27:14.850 INFO:teuthology.orchestra.run.smithi092.stdout: args: 2022-12-03T01:27:14.850 INFO:teuthology.orchestra.run.smithi092.stdout: - -f 2022-12-03T01:27:14.850 INFO:teuthology.orchestra.run.smithi092.stdout: - /etc/kube-flannel/cni-conf.json 2022-12-03T01:27:14.851 INFO:teuthology.orchestra.run.smithi092.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-12-03T01:27:14.851 INFO:teuthology.orchestra.run.smithi092.stdout: volumeMounts: 2022-12-03T01:27:14.851 INFO:teuthology.orchestra.run.smithi092.stdout: - name: cni 2022-12-03T01:27:14.851 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /etc/cni/net.d 2022-12-03T01:27:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: - name: flannel-cfg 2022-12-03T01:27:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /etc/kube-flannel/ 2022-12-03T01:27:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: containers: 2022-12-03T01:27:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: - name: kube-flannel 2022-12-03T01:27:14.852 INFO:teuthology.orchestra.run.smithi092.stdout: #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply) 2022-12-03T01:27:14.853 INFO:teuthology.orchestra.run.smithi092.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2 2022-12-03T01:27:14.853 INFO:teuthology.orchestra.run.smithi092.stdout: command: 2022-12-03T01:27:14.853 INFO:teuthology.orchestra.run.smithi092.stdout: - /opt/bin/flanneld 2022-12-03T01:27:14.853 INFO:teuthology.orchestra.run.smithi092.stdout: args: 2022-12-03T01:27:14.854 INFO:teuthology.orchestra.run.smithi092.stdout: - --ip-masq 2022-12-03T01:27:14.854 INFO:teuthology.orchestra.run.smithi092.stdout: - --kube-subnet-mgr 2022-12-03T01:27:14.854 INFO:teuthology.orchestra.run.smithi092.stdout: resources: 2022-12-03T01:27:14.854 INFO:teuthology.orchestra.run.smithi092.stdout: requests: 2022-12-03T01:27:14.855 INFO:teuthology.orchestra.run.smithi092.stdout: cpu: "100m" 2022-12-03T01:27:14.855 INFO:teuthology.orchestra.run.smithi092.stdout: memory: "50Mi" 2022-12-03T01:27:14.855 INFO:teuthology.orchestra.run.smithi092.stdout: limits: 2022-12-03T01:27:14.855 INFO:teuthology.orchestra.run.smithi092.stdout: cpu: "100m" 2022-12-03T01:27:14.855 INFO:teuthology.orchestra.run.smithi092.stdout: memory: "50Mi" 2022-12-03T01:27:14.856 INFO:teuthology.orchestra.run.smithi092.stdout: securityContext: 2022-12-03T01:27:14.856 INFO:teuthology.orchestra.run.smithi092.stdout: privileged: false 2022-12-03T01:27:14.856 INFO:teuthology.orchestra.run.smithi092.stdout: capabilities: 2022-12-03T01:27:14.856 INFO:teuthology.orchestra.run.smithi092.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-12-03T01:27:14.857 INFO:teuthology.orchestra.run.smithi092.stdout: env: 2022-12-03T01:27:14.857 INFO:teuthology.orchestra.run.smithi092.stdout: - name: POD_NAME 2022-12-03T01:27:14.857 INFO:teuthology.orchestra.run.smithi092.stdout: valueFrom: 2022-12-03T01:27:14.857 INFO:teuthology.orchestra.run.smithi092.stdout: fieldRef: 2022-12-03T01:27:14.857 INFO:teuthology.orchestra.run.smithi092.stdout: fieldPath: metadata.name 2022-12-03T01:27:14.858 INFO:teuthology.orchestra.run.smithi092.stdout: - name: POD_NAMESPACE 2022-12-03T01:27:14.858 INFO:teuthology.orchestra.run.smithi092.stdout: valueFrom: 2022-12-03T01:27:14.858 INFO:teuthology.orchestra.run.smithi092.stdout: fieldRef: 2022-12-03T01:27:14.858 INFO:teuthology.orchestra.run.smithi092.stdout: fieldPath: metadata.namespace 2022-12-03T01:27:14.859 INFO:teuthology.orchestra.run.smithi092.stdout: - name: EVENT_QUEUE_DEPTH 2022-12-03T01:27:14.859 INFO:teuthology.orchestra.run.smithi092.stdout: value: "5000" 2022-12-03T01:27:14.859 INFO:teuthology.orchestra.run.smithi092.stdout: volumeMounts: 2022-12-03T01:27:14.859 INFO:teuthology.orchestra.run.smithi092.stdout: - name: run 2022-12-03T01:27:14.860 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /run/flannel 2022-12-03T01:27:14.860 INFO:teuthology.orchestra.run.smithi092.stdout: - name: flannel-cfg 2022-12-03T01:27:14.861 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /etc/kube-flannel/ 2022-12-03T01:27:14.861 INFO:teuthology.orchestra.run.smithi092.stdout: - name: xtables-lock 2022-12-03T01:27:14.861 INFO:teuthology.orchestra.run.smithi092.stdout: mountPath: /run/xtables.lock 2022-12-03T01:27:14.861 INFO:teuthology.orchestra.run.smithi092.stdout: volumes: 2022-12-03T01:27:14.861 INFO:teuthology.orchestra.run.smithi092.stdout: - name: run 2022-12-03T01:27:14.862 INFO:teuthology.orchestra.run.smithi092.stdout: hostPath: 2022-12-03T01:27:14.862 INFO:teuthology.orchestra.run.smithi092.stdout: path: /run/flannel 2022-12-03T01:27:14.862 INFO:teuthology.orchestra.run.smithi092.stdout: - name: cni-plugin 2022-12-03T01:27:14.862 INFO:teuthology.orchestra.run.smithi092.stdout: hostPath: 2022-12-03T01:27:14.863 INFO:teuthology.orchestra.run.smithi092.stdout: path: /opt/cni/bin 2022-12-03T01:27:14.863 INFO:teuthology.orchestra.run.smithi092.stdout: - name: cni 2022-12-03T01:27:14.863 INFO:teuthology.orchestra.run.smithi092.stdout: hostPath: 2022-12-03T01:27:14.863 INFO:teuthology.orchestra.run.smithi092.stdout: path: /etc/cni/net.d 2022-12-03T01:27:14.864 INFO:teuthology.orchestra.run.smithi092.stdout: - name: flannel-cfg 2022-12-03T01:27:14.864 INFO:teuthology.orchestra.run.smithi092.stdout: configMap: 2022-12-03T01:27:14.864 INFO:teuthology.orchestra.run.smithi092.stdout: name: kube-flannel-cfg 2022-12-03T01:27:14.864 INFO:teuthology.orchestra.run.smithi092.stdout: - name: xtables-lock 2022-12-03T01:27:14.864 INFO:teuthology.orchestra.run.smithi092.stdout: hostPath: 2022-12-03T01:27:14.865 INFO:teuthology.orchestra.run.smithi092.stdout: path: /run/xtables.lock 2022-12-03T01:27:14.865 INFO:teuthology.orchestra.run.smithi092.stdout: type: FileOrCreate 2022-12-03T01:27:14.866 INFO:teuthology.orchestra.run.smithi092.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 4591 100 4591 0 0 28874 0 --:--:-- --:--:-- --:--:-- 28874 2022-12-03T01:27:14.888 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-flannel', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-12-03T01:27:14.889 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-flannel', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.250.216.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-12-03T01:27:14.900 DEBUG:tasks.kubeadm:Flannel: apiVersion: v1 kind: Namespace metadata: labels: pod-security.kubernetes.io/enforce: privileged name: kube-flannel --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - get - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-flannel --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-flannel --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.250.216.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-flannel --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-flannel spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: '5000' image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-12-03T01:27:14.901 DEBUG:teuthology.orchestra.run.smithi092:> kubectl apply -f - 2022-12-03T01:27:15.592 INFO:teuthology.orchestra.run.smithi092.stdout:namespace/kube-flannel created 2022-12-03T01:27:15.644 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-12-03T01:27:15.671 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-12-03T01:27:15.684 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/flannel created 2022-12-03T01:27:15.697 INFO:teuthology.orchestra.run.smithi092.stdout:configmap/kube-flannel-cfg created 2022-12-03T01:27:15.723 INFO:teuthology.orchestra.run.smithi092.stdout:daemonset.apps/kube-flannel-ds created 2022-12-03T01:27:15.727 INFO:tasks.kubeadm:Scanning for scratch devices 2022-12-03T01:27:15.727 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:15.728 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:15.733 INFO:tasks.kubeadm:Scratch devices on smithi092: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:15.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:27:15.803 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records in 2022-12-03T01:27:15.804 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2022-12-03T01:27:15.804 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148727 s, 705 MB/s 2022-12-03T01:27:15.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:27:15.874 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records in 2022-12-03T01:27:15.874 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2022-12-03T01:27:15.875 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0172337 s, 608 MB/s 2022-12-03T01:27:15.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:27:15.946 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records in 2022-12-03T01:27:15.947 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2022-12-03T01:27:15.947 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0165678 s, 633 MB/s 2022-12-03T01:27:15.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:27:16.018 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records in 2022-12-03T01:27:16.018 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2022-12-03T01:27:16.018 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0179553 s, 584 MB/s 2022-12-03T01:27:16.020 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:16.020 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:16.030 INFO:tasks.kubeadm:Scratch devices on smithi121: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:16.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:27:16.105 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records in 2022-12-03T01:27:16.105 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-12-03T01:27:16.106 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0180279 s, 582 MB/s 2022-12-03T01:27:16.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:27:16.181 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records in 2022-12-03T01:27:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-12-03T01:27:16.182 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0185351 s, 566 MB/s 2022-12-03T01:27:16.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:27:16.262 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records in 2022-12-03T01:27:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-12-03T01:27:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0192796 s, 544 MB/s 2022-12-03T01:27:16.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:27:16.340 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records in 2022-12-03T01:27:16.341 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2022-12-03T01:27:16.341 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175222 s, 598 MB/s 2022-12-03T01:27:16.342 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:16.343 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:16.351 INFO:tasks.kubeadm:Scratch devices on smithi138: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:16.351 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-12-03T01:27:16.426 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records in 2022-12-03T01:27:16.426 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2022-12-03T01:27:16.427 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0174133 s, 602 MB/s 2022-12-03T01:27:16.428 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-12-03T01:27:16.502 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records in 2022-12-03T01:27:16.502 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2022-12-03T01:27:16.502 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0176501 s, 594 MB/s 2022-12-03T01:27:16.503 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-12-03T01:27:16.578 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records in 2022-12-03T01:27:16.579 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2022-12-03T01:27:16.579 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0175229 s, 598 MB/s 2022-12-03T01:27:16.581 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-12-03T01:27:16.652 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records in 2022-12-03T01:27:16.653 INFO:teuthology.orchestra.run.smithi138.stderr:10+0 records out 2022-12-03T01:27:16.653 INFO:teuthology.orchestra.run.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159676 s, 657 MB/s 2022-12-03T01:27:16.672 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-12-03T01:27:16.672 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi092-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi092 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi092-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi092 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi092-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi092 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi092-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi092 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi121-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi121 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi121-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi121 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi121-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi121 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi121-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi121 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi138-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi138 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi138-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi138 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi138-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi138 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi138-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi138 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-12-03T01:27:16.673 DEBUG:teuthology.orchestra.run.smithi092:> kubectl create -f - 2022-12-03T01:27:16.852 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi092-lv-1 created 2022-12-03T01:27:16.876 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi092-lv-2 created 2022-12-03T01:27:16.909 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi092-lv-3 created 2022-12-03T01:27:16.941 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi092-lv-4 created 2022-12-03T01:27:17.033 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi121-lv-1 created 2022-12-03T01:27:17.096 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi121-lv-2 created 2022-12-03T01:27:17.126 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi121-lv-3 created 2022-12-03T01:27:17.209 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi121-lv-4 created 2022-12-03T01:27:17.242 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi138-lv-1 created 2022-12-03T01:27:17.352 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi138-lv-2 created 2022-12-03T01:27:17.384 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi138-lv-3 created 2022-12-03T01:27:17.444 INFO:teuthology.orchestra.run.smithi092.stdout:persistentvolume/smithi138-lv-4 created 2022-12-03T01:27:17.519 INFO:teuthology.orchestra.run.smithi092.stdout:storageclass.storage.k8s.io/scratch created 2022-12-03T01:27:17.525 DEBUG:teuthology.orchestra.run.smithi092:> kubectl taint node smithi092 node-role.kubernetes.io/master- || true 2022-12-03T01:27:17.601 INFO:teuthology.orchestra.run.smithi092.stderr:error: taint "node-role.kubernetes.io/master" not found 2022-12-03T01:27:17.603 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-12-03T01:27:17.604 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-12-03T01:27:17.618 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-12-03T01:27:17.619 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:17.619 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:17.627 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:17.627 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2022-12-03T01:27:17.677 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:27:17.678 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:17.678 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 493 Links: 1 2022-12-03T01:27:17.678 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:17.679 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-12-03 01:27:15.806763840 +0000 2022-12-03T01:27:17.679 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-12-03 01:27:15.806763840 +0000 2022-12-03T01:27:17.679 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-12-03 01:27:15.806763840 +0000 2022-12-03T01:27:17.679 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-12-03T01:27:17.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:27:17.732 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-12-03T01:27:17.732 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-12-03T01:27:17.733 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000461607 s, 1.1 MB/s 2022-12-03T01:27:17.735 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:27:17.791 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2022-12-03T01:27:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:27:17.842 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:17.842 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-12-03T01:27:17.843 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:17.843 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-12-03 01:27:15.878763838 +0000 2022-12-03T01:27:17.843 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-12-03 01:27:15.878763838 +0000 2022-12-03T01:27:17.843 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-12-03 01:27:15.878763838 +0000 2022-12-03T01:27:17.844 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-12-03T01:27:17.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:27:17.902 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-12-03T01:27:17.902 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-12-03T01:27:17.903 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00043707 s, 1.2 MB/s 2022-12-03T01:27:17.904 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:27:17.959 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2022-12-03T01:27:18.014 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:27:18.015 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:18.015 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-12-03T01:27:18.015 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:18.015 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-12-03 01:27:15.950763837 +0000 2022-12-03T01:27:18.016 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-12-03 01:27:15.950763837 +0000 2022-12-03T01:27:18.016 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-12-03 01:27:15.950763837 +0000 2022-12-03T01:27:18.016 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-12-03T01:27:18.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:27:18.072 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-12-03T01:27:18.073 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-12-03T01:27:18.073 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000417131 s, 1.2 MB/s 2022-12-03T01:27:18.075 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:27:18.127 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2022-12-03T01:27:18.178 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:27:18.179 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:18.179 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-12-03T01:27:18.179 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:18.179 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2022-12-03 01:27:16.022763835 +0000 2022-12-03T01:27:18.179 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2022-12-03 01:27:16.022763835 +0000 2022-12-03T01:27:18.180 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2022-12-03 01:27:16.022763835 +0000 2022-12-03T01:27:18.180 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2022-12-03T01:27:18.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:27:18.233 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2022-12-03T01:27:18.233 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2022-12-03T01:27:18.233 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000455309 s, 1.1 MB/s 2022-12-03T01:27:18.235 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:27:18.290 DEBUG:teuthology.orchestra.run.smithi092:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:27:18.547 INFO:teuthology.orchestra.run.smithi092.stdout:loop 2022-12-03T01:27:18.548 INFO:tasks.nvme_loop:Connecting nvme_loop smithi092:/dev/vg_nvme/lv_1... 2022-12-03T01:27:18.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:27:18.572 INFO:teuthology.orchestra.run.smithi092.stdout:1 2022-12-03T01:27:18.593 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:27:18.612 INFO:tasks.nvme_loop:Connecting nvme_loop smithi092:/dev/vg_nvme/lv_2... 2022-12-03T01:27:18.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:27:18.677 INFO:teuthology.orchestra.run.smithi092.stdout:1 2022-12-03T01:27:18.701 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:27:18.718 INFO:tasks.nvme_loop:Connecting nvme_loop smithi092:/dev/vg_nvme/lv_3... 2022-12-03T01:27:18.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:27:18.782 INFO:teuthology.orchestra.run.smithi092.stdout:1 2022-12-03T01:27:18.809 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:27:18.830 INFO:tasks.nvme_loop:Connecting nvme_loop smithi092:/dev/vg_nvme/lv_4... 2022-12-03T01:27:18.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:27:18.891 INFO:teuthology.orchestra.run.smithi092.stdout:1 2022-12-03T01:27:18.913 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:27:18.930 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:18.931 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:18.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo nvme list 2022-12-03T01:27:19.041 INFO:teuthology.orchestra.run.smithi092.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:27:19.041 INFO:teuthology.orchestra.run.smithi092.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:27:19.041 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/nvme0n1 PHFT620400QM400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-12-03T01:27:19.042 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/nvme1n1 875b43cd25a6437c5662 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:19.042 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/nvme2n1 308f3961c38da7634e99 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:19.042 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/nvme3n1 5c7e1578789225b44a7c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:19.043 INFO:teuthology.orchestra.run.smithi092.stdout:/dev/nvme4n1 f988efbd2420a5280a95 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:19.045 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:27:19.045 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:19.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/scratch_devs 2022-12-03T01:27:19.118 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:19.118 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:19.126 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:19.127 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2022-12-03T01:27:19.176 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:27:19.177 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 493 Links: 1 2022-12-03T01:27:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-12-03 01:27:16.109991555 +0000 2022-12-03T01:27:19.179 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-12-03 01:27:16.109991555 +0000 2022-12-03T01:27:19.179 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-12-03 01:27:16.109991555 +0000 2022-12-03T01:27:19.180 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-12-03T01:27:19.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:27:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-12-03T01:27:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-12-03T01:27:19.235 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000599703 s, 854 kB/s 2022-12-03T01:27:19.236 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:27:19.289 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2022-12-03T01:27:19.344 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:27:19.344 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:19.344 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-12-03T01:27:19.345 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:19.345 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-12-03 01:27:16.185991554 +0000 2022-12-03T01:27:19.345 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-12-03 01:27:16.185991554 +0000 2022-12-03T01:27:19.345 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-12-03 01:27:16.185991554 +0000 2022-12-03T01:27:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-12-03T01:27:19.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:27:19.406 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-12-03T01:27:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-12-03T01:27:19.407 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000761837 s, 672 kB/s 2022-12-03T01:27:19.408 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:27:19.461 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2022-12-03T01:27:19.511 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:27:19.512 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 505 Links: 1 2022-12-03T01:27:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-12-03 01:27:16.269991552 +0000 2022-12-03T01:27:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-12-03 01:27:16.269991552 +0000 2022-12-03T01:27:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-12-03 01:27:16.269991552 +0000 2022-12-03T01:27:19.514 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-12-03T01:27:19.514 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:27:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-12-03T01:27:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-12-03T01:27:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000549712 s, 931 kB/s 2022-12-03T01:27:19.573 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:27:19.625 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2022-12-03T01:27:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:27:19.675 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:19.676 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-12-03T01:27:19.676 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:19.676 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2022-12-03 01:27:16.345991550 +0000 2022-12-03T01:27:19.676 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2022-12-03 01:27:16.345991550 +0000 2022-12-03T01:27:19.676 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2022-12-03 01:27:16.345991550 +0000 2022-12-03T01:27:19.677 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2022-12-03T01:27:19.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:27:19.729 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2022-12-03T01:27:19.730 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2022-12-03T01:27:19.730 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000627906 s, 815 kB/s 2022-12-03T01:27:19.732 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:27:19.785 DEBUG:teuthology.orchestra.run.smithi121:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:27:19.924 INFO:teuthology.orchestra.run.smithi121.stdout:loop 2022-12-03T01:27:19.925 INFO:tasks.nvme_loop:Connecting nvme_loop smithi121:/dev/vg_nvme/lv_1... 2022-12-03T01:27:19.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:27:19.945 INFO:teuthology.orchestra.run.smithi121.stdout:1 2022-12-03T01:27:19.969 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:27:19.993 INFO:tasks.nvme_loop:Connecting nvme_loop smithi121:/dev/vg_nvme/lv_2... 2022-12-03T01:27:19.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:27:20.057 INFO:teuthology.orchestra.run.smithi121.stdout:1 2022-12-03T01:27:20.086 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:27:20.112 INFO:tasks.nvme_loop:Connecting nvme_loop smithi121:/dev/vg_nvme/lv_3... 2022-12-03T01:27:20.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:27:20.180 INFO:teuthology.orchestra.run.smithi121.stdout:1 2022-12-03T01:27:20.204 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:27:20.228 INFO:tasks.nvme_loop:Connecting nvme_loop smithi121:/dev/vg_nvme/lv_4... 2022-12-03T01:27:20.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:27:20.292 INFO:teuthology.orchestra.run.smithi121.stdout:1 2022-12-03T01:27:20.315 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:27:20.340 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:20.342 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:20.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo nvme list 2022-12-03T01:27:20.455 INFO:teuthology.orchestra.run.smithi121.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:27:20.456 INFO:teuthology.orchestra.run.smithi121.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:27:20.456 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/nvme0n1 PHFT620400ZF400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101F0 2022-12-03T01:27:20.456 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/nvme1n1 9e6245d481b33344dab5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:20.456 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/nvme2n1 88ab292d4e2f655ea267 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:20.457 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/nvme3n1 e04c6cfb3d45985a6f93 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:20.457 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/nvme4n1 a53c35454c3919cb2ecf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:20.458 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:27:20.458 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:27:20.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/scratch_devs 2022-12-03T01:27:20.516 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:20.516 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:20.524 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-12-03T01:27:20.525 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2022-12-03T01:27:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-12-03T01:27:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:20.574 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 494 Links: 1 2022-12-03T01:27:20.574 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:20.574 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-12-03 01:27:16.430979660 +0000 2022-12-03T01:27:20.575 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-12-03 01:27:16.426979660 +0000 2022-12-03T01:27:20.575 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-12-03 01:27:16.426979660 +0000 2022-12-03T01:27:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-12-03T01:27:20.576 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2022-12-03T01:27:20.635 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-12-03T01:27:20.635 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-12-03T01:27:20.636 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000561316 s, 912 kB/s 2022-12-03T01:27:20.637 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2022-12-03T01:27:20.687 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2022-12-03T01:27:20.740 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-12-03T01:27:20.741 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:20.741 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-12-03T01:27:20.741 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:20.742 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-12-03 01:27:16.506979658 +0000 2022-12-03T01:27:20.742 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-12-03 01:27:16.506979658 +0000 2022-12-03T01:27:20.742 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-12-03 01:27:16.506979658 +0000 2022-12-03T01:27:20.742 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-12-03T01:27:20.743 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2022-12-03T01:27:20.796 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-12-03T01:27:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-12-03T01:27:20.797 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000545866 s, 938 kB/s 2022-12-03T01:27:20.799 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2022-12-03T01:27:20.850 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2022-12-03T01:27:20.901 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-12-03T01:27:20.901 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:20.902 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-12-03T01:27:20.902 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:20.902 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-12-03 01:27:16.582979656 +0000 2022-12-03T01:27:20.902 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-12-03 01:27:16.582979656 +0000 2022-12-03T01:27:20.902 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-12-03 01:27:16.582979656 +0000 2022-12-03T01:27:20.903 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-12-03T01:27:20.903 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2022-12-03T01:27:20.959 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-12-03T01:27:20.960 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-12-03T01:27:20.960 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000543151 s, 943 kB/s 2022-12-03T01:27:20.961 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2022-12-03T01:27:21.011 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2022-12-03T01:27:21.066 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-12-03T01:27:21.066 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-12-03T01:27:21.066 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 512 Links: 1 2022-12-03T01:27:21.066 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-12-03T01:27:21.067 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2022-12-03 01:27:16.658979655 +0000 2022-12-03T01:27:21.067 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2022-12-03 01:27:16.658979655 +0000 2022-12-03T01:27:21.067 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2022-12-03 01:27:16.658979655 +0000 2022-12-03T01:27:21.067 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2022-12-03T01:27:21.068 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2022-12-03T01:27:21.130 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2022-12-03T01:27:21.131 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2022-12-03T01:27:21.132 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000566177 s, 904 kB/s 2022-12-03T01:27:21.133 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2022-12-03T01:27:21.184 DEBUG:teuthology.orchestra.run.smithi138:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-12-03T01:27:21.353 INFO:teuthology.orchestra.run.smithi138.stdout:loop 2022-12-03T01:27:21.355 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_1... 2022-12-03T01:27:21.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-12-03T01:27:21.377 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-12-03T01:27:21.399 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_11 2022-12-03T01:27:21.418 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_2... 2022-12-03T01:27:21.418 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-12-03T01:27:21.485 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-12-03T01:27:21.506 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_21 2022-12-03T01:27:21.528 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_3... 2022-12-03T01:27:21.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-12-03T01:27:21.593 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-12-03T01:27:21.619 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_31 2022-12-03T01:27:21.644 INFO:tasks.nvme_loop:Connecting nvme_loop smithi138:/dev/vg_nvme/lv_4... 2022-12-03T01:27:21.645 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-12-03T01:27:21.712 INFO:teuthology.orchestra.run.smithi138.stdout:1 2022-12-03T01:27:21.736 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/vg_nvme/lv_41 2022-12-03T01:27:21.757 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:21.757 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:27:21.808 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme list 2022-12-03T01:27:21.865 INFO:teuthology.orchestra.run.smithi138.stdout:Node SN Model Namespace Usage Format FW Rev 2022-12-03T01:27:21.865 INFO:teuthology.orchestra.run.smithi138.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-12-03T01:27:21.865 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme0n1 CVFT6161003C400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-12-03T01:27:21.866 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme1n1 76791a42560e9159fe1a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:21.866 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme2n1 719d7b50d7ce0e03c1e5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:21.866 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme3n1 546c7c8443a3165a5701 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:21.866 INFO:teuthology.orchestra.run.smithi138.stdout:/dev/nvme4n1 1f1a29fa8de15b07b9ec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-5 2022-12-03T01:27:21.867 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-12-03T01:27:21.868 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:27:21.868 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/scratch_devs 2022-12-03T01:27:21.931 INFO:teuthology.run_tasks:Running task rook... 2022-12-03T01:27:22.031 INFO:tasks.rook:Rook start 2022-12-03T01:27:22.032 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '48d203d9b836cc224eb4085f6198d948c6fa1bb1', 'rook_image': 'rook/ceph:master'} 2022-12-03T01:27:22.032 INFO:tasks.rook:Ceph image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1 2022-12-03T01:27:22.033 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-12-03T01:27:22.033 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-12-03T01:27:22.043 INFO:teuthology.orchestra.run.smithi092.stderr:Cloning into 'rook'... 2022-12-03T01:27:25.260 INFO:tasks.rook:/home/teuthworker 2022-12-03T01:27:25.261 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/sbin:/usr/sbin ip addr show 2022-12-03T01:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-12-03T01:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-12-03T01:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: inet 127.0.0.1/8 scope host lo 2022-12-03T01:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 ::1/128 scope host 2022-12-03T01:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b2 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f0 2022-12-03T01:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:d8:a1:b3 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: altname enp7s0f1 2022-12-03T01:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-12-03T01:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:44 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-12-03T01:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 0c:c4:7a:8f:cd:45 brd ff:ff:ff:ff:ff:ff 2022-12-03T01:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: inet 172.21.15.92/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-12-03T01:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft 42734sec preferred_lft 42734sec 2022-12-03T01:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2022-12-03T01:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-12-03T01:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: link/ether 02:42:8f:6e:d9:3c brd ff:ff:ff:ff:ff:ff 2022-12-03T01:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-12-03T01:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: valid_lft forever preferred_lft forever 2022-12-03T01:27:25.319 DEBUG:teuthology.orchestra.run.smithi092:> which systemctl 2022-12-03T01:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/systemctl 2022-12-03T01:27:25.357 DEBUG:teuthology.orchestra.run.smithi092:> true 2022-12-03T01:27:25.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2022-12-03T01:27:25.488 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2022-12-03T01:27:25.493 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-12-03T01:27:25.494 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'is_mounted', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-12-03T01:27:25.543 DEBUG:teuthology.orchestra.remote:smithi092:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:27:25.553 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:25.554 DEBUG:teuthology.orchestra.run.smithi092:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-12-03T01:27:25.561 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-12-03T01:27:25.612 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:27:25.613 DEBUG:teuthology.orchestra.run.smithi092:> dd of=operator.yaml 2022-12-03T01:27:25.620 INFO:tasks.rook:Deploying operator 2022-12-03T01:27:25.669 DEBUG:teuthology.orchestra.remote:smithi092:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:27:25.692 DEBUG:teuthology.orchestra.remote:smithi092:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:27:25.706 DEBUG:teuthology.orchestra.run.smithi092:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-12-03T01:27:26.096 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created 2022-12-03T01:27:26.656 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-12-03T01:27:27.179 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-12-03T01:27:27.419 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-12-03T01:27:27.502 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-12-03T01:27:27.668 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-12-03T01:27:28.039 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-12-03T01:27:28.240 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-12-03T01:27:28.287 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-12-03T01:27:28.464 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-12-03T01:27:28.549 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-12-03T01:27:28.617 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-12-03T01:27:28.766 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-12-03T01:27:28.817 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-12-03T01:27:28.968 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-12-03T01:27:29.361 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-12-03T01:27:29.586 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-12-03T01:27:29.765 INFO:teuthology.orchestra.run.smithi092.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-12-03T01:27:29.833 INFO:teuthology.orchestra.run.smithi092.stdout:namespace/rook-ceph created 2022-12-03T01:27:29.929 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-12-03T01:27:29.944 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-12-03T01:27:30.003 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:27:30.020 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-12-03T01:27:30.082 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-12-03T01:27:30.095 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-12-03T01:27:30.102 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-12-03T01:27:30.126 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-12-03T01:27:30.136 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-12-03T01:27:30.144 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:27:30.152 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:27:30.169 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-12-03T01:27:30.177 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:27:30.223 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-12-03T01:27:30.236 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-12-03T01:27:30.244 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-12-03T01:27:30.253 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-12-03T01:27:30.261 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:27:30.269 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:27:30.293 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-12-03T01:27:30.302 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-12-03T01:27:30.310 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-12-03T01:27:30.319 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-12-03T01:27:30.348 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-12-03T01:27:30.386 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:27:30.402 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-12-03T01:27:30.426 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-12-03T01:27:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:27:30.474 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-12-03T01:27:30.486 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-12-03T01:27:30.494 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-12-03T01:27:30.502 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-12-03T01:27:30.526 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-12-03T01:27:30.536 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-12-03T01:27:30.544 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-12-03T01:27:30.553 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-12-03T01:27:30.562 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-12-03T01:27:30.570 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-12-03T01:27:30.608 INFO:teuthology.orchestra.run.smithi092.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-12-03T01:27:30.621 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-12-03T01:27:30.628 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-mgr created 2022-12-03T01:27:30.636 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-osd created 2022-12-03T01:27:30.644 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-purge-osd created 2022-12-03T01:27:30.653 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-rgw created 2022-12-03T01:27:30.661 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-ceph-system created 2022-12-03T01:27:30.669 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-12-03T01:27:30.693 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-12-03T01:27:30.702 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-12-03T01:27:30.801 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-12-03T01:27:30.852 INFO:teuthology.orchestra.run.smithi092.stdout:configmap/rook-ceph-operator-config created 2022-12-03T01:27:30.886 INFO:teuthology.orchestra.run.smithi092.stdout:deployment.apps/rook-ceph-operator created 2022-12-03T01:27:30.893 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:27:31.005 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:27:31.006 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 Pending 0 1s 2022-12-03T01:27:41.010 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:27:41.095 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:27:41.096 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 Pending 0 11s 2022-12-03T01:27:51.098 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:27:51.172 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:27:51.172 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 ContainerCreating 0 21s 2022-12-03T01:28:01.175 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:28:01.250 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:01.250 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 ContainerCreating 0 31s 2022-12-03T01:28:11.253 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:28:11.323 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:11.324 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 ContainerCreating 0 41s 2022-12-03T01:28:21.327 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:28:21.403 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:21.404 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 0/1 ContainerCreating 0 51s 2022-12-03T01:28:31.407 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-12-03T01:28:31.484 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:31.484 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 61s 2022-12-03T01:28:31.486 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph logs -f rook-ceph-operator-5ddcc89457-mhcgp 2022-12-03T01:28:31.501 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:28:31.501 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:28:31.554 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:28:31.555 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:28:31.564 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:28:31.565 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2022-12-03T01:28:31.576 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-12-03T01:28:31.576 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-12-03T01:28:31.576 INFO:tasks.rook: override: [mon] debug mon = 20 2022-12-03T01:28:31.577 INFO:tasks.rook: override: [mon] debug ms = 1 2022-12-03T01:28:31.577 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-12-03T01:28:31.577 INFO:tasks.rook: override: [osd] debug ms = 1 2022-12-03T01:28:31.578 INFO:tasks.rook: override: [osd] debug osd = 20 2022-12-03T01:28:31.579 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-12-03T01:28:31.584 DEBUG:teuthology.orchestra.run.smithi092:> kubectl create -f - 2022-12-03T01:28:31.590 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:20.944881 I | rookcmd: starting Rook v1.10.0-alpha.0.416.gfe1ef6c22 with arguments '/usr/local/bin/rook ceph operator' 2022-12-03T01:28:31.591 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:20.944975 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-12-03T01:28:31.591 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:20.944977 I | cephcmd: starting Rook-Ceph operator 2022-12-03T01:28:31.591 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.076987 I | cephcmd: base ceph version inside the rook operator image is "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)" 2022-12-03T01:28:31.591 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.084272 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-12-03T01:28:31.592 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.084290 I | operator: watching all namespaces for Ceph CRs 2022-12-03T01:28:31.592 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.084361 I | operator: setting up schemes 2022-12-03T01:28:31.592 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.087507 I | operator: setting up the controller-runtime manager 2022-12-03T01:28:31.592 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.097280 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-12-03T01:28:31.593 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099462 I | operator: failed to get cert manager 2022-12-03T01:28:31.593 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099501 I | ceph-cluster-controller: successfully started 2022-12-03T01:28:31.594 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099574 I | ceph-cluster-controller: enabling hotplug orchestration 2022-12-03T01:28:31.594 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099592 I | ceph-crashcollector-controller: successfully started 2022-12-03T01:28:31.594 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099619 I | ceph-block-pool-controller: successfully started 2022-12-03T01:28:31.594 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099638 I | ceph-object-store-user-controller: successfully started 2022-12-03T01:28:31.595 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099656 I | ceph-object-realm-controller: successfully started 2022-12-03T01:28:31.595 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099668 I | ceph-object-zonegroup-controller: successfully started 2022-12-03T01:28:31.595 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099690 I | ceph-object-zone-controller: successfully started 2022-12-03T01:28:31.595 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099831 I | ceph-object-controller: successfully started 2022-12-03T01:28:31.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099862 I | ceph-file-controller: successfully started 2022-12-03T01:28:31.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099904 I | ceph-nfs-controller: successfully started 2022-12-03T01:28:31.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099929 I | ceph-rbd-mirror-controller: successfully started 2022-12-03T01:28:31.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.099969 I | ceph-client-controller: successfully started 2022-12-03T01:28:31.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100001 I | ceph-filesystem-mirror-controller: successfully started 2022-12-03T01:28:31.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100038 I | operator: rook-ceph-operator-config-controller successfully started 2022-12-03T01:28:31.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100061 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-12-03T01:28:31.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100085 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-12-03T01:28:31.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100100 I | ceph-bucket-topic: successfully started 2022-12-03T01:28:31.598 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100113 I | ceph-bucket-notification: successfully started 2022-12-03T01:28:31.598 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100135 I | ceph-bucket-notification: successfully started 2022-12-03T01:28:31.598 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100148 I | ceph-fs-subvolumegroup-controller: successfully started 2022-12-03T01:28:31.598 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.100162 I | blockpool-rados-namespace-controller: successfully started 2022-12-03T01:28:31.599 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.101048 I | operator: starting the controller-runtime manager 2022-12-03T01:28:31.599 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.204912 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-12-03T01:28:31.599 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.204940 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-12-03T01:28:31.600 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.204955 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-12-03T01:28:31.600 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.205634 I | ceph-csi: CSI Ceph RBD driver disabled 2022-12-03T01:28:31.602 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.205654 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-12-03T01:28:31.602 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.208493 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap) 2022-12-03T01:28:31.602 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.208515 I | operator: rook-ceph-operator-config-controller done reconciling 2022-12-03T01:28:31.603 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.208559 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-12-03T01:28:31.603 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.215085 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-12-03T01:28:31.603 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.215107 I | ceph-csi: CSI CephFS driver disabled 2022-12-03T01:28:31.604 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.215121 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-12-03T01:28:31.607 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.217048 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-12-03T01:28:31.607 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.222862 I | ceph-csi: successfully removed CSI CephFS driver 2022-12-03T01:28:31.608 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.222877 I | ceph-csi: CSI NFS driver disabled 2022-12-03T01:28:31.608 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.222883 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-12-03T01:28:31.608 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.224746 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-12-03T01:28:31.609 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:21.230585 I | ceph-csi: successfully removed CSI NFS driver 2022-12-03T01:28:32.399 INFO:teuthology.orchestra.run.smithi092.stdout:configmap/rook-config-override created 2022-12-03T01:28:32.406 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true storage: storageClassDeviceSets: - count: 12 name: scratch portable: false volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: scratch volumeMode: Block 2022-12-03T01:28:32.407 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:28:32.407 DEBUG:teuthology.orchestra.run.smithi092:> dd of=cluster.yaml 2022-12-03T01:28:32.414 DEBUG:teuthology.orchestra.run.smithi092:> kubectl create -f cluster.yaml 2022-12-03T01:28:32.733 INFO:teuthology.orchestra.run.smithi092.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-12-03T01:28:32.734 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.730833 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-12-03T01:28:32.735 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.732232 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:28:32.743 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.745178 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-12-03T01:28:32.755 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.752152 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-12-03T01:28:32.756 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.752206 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-12-03T01:28:32.763 DEBUG:teuthology.orchestra.remote:smithi092:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:28:32.780 DEBUG:teuthology.orchestra.run.smithi092:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-12-03T01:28:32.812 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.814820 I | ceph-spec: detecting the ceph image version for image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1... 2022-12-03T01:28:32.831 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829280 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-12-03T01:28:32.832 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829302 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-12-03T01:28:32.832 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829311 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap) 2022-12-03T01:28:32.833 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829336 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-12-03T01:28:32.833 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829345 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-12-03T01:28:32.834 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829354 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:28:32.834 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829365 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-12-03T01:28:32.835 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829382 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap) 2022-12-03T01:28:32.835 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829393 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-12-03T01:28:32.836 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829405 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-12-03T01:28:32.837 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829414 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-12-03T01:28:32.837 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829425 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-12-03T01:28:32.838 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829435 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-12-03T01:28:32.838 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829446 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-12-03T01:28:32.838 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829456 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-12-03T01:28:32.839 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829468 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-12-03T01:28:32.840 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829479 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-12-03T01:28:32.840 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829490 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-12-03T01:28:32.840 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829500 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap) 2022-12-03T01:28:32.841 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829510 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap) 2022-12-03T01:28:32.841 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829520 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap) 2022-12-03T01:28:32.842 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829529 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-12-03T01:28:32.842 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829544 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-12-03T01:28:32.842 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829553 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-12-03T01:28:32.843 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829564 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap) 2022-12-03T01:28:32.843 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829570 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-12-03T01:28:32.843 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829575 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap) 2022-12-03T01:28:32.843 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829581 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap) 2022-12-03T01:28:32.844 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829587 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default) 2022-12-03T01:28:32.844 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829597 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:28:32.844 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829609 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:28:32.844 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829620 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-12-03T01:28:32.845 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829625 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default) 2022-12-03T01:28:32.845 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829631 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-12-03T01:28:32.845 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829636 I | ceph-csi: Kubernetes version is 1.25 2022-12-03T01:28:32.846 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829643 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.6.0" (default) 2022-12-03T01:28:32.846 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829649 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-12-03T01:28:32.846 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.829654 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default) 2022-12-03T01:28:32.848 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834571 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-12-03T01:28:32.849 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834595 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.7.2" (default) 2022-12-03T01:28:32.849 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834610 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" (default) 2022-12-03T01:28:32.849 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834622 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v3.3.0" (default) 2022-12-03T01:28:32.849 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834635 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.0.0" (default) 2022-12-03T01:28:32.850 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834648 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" (default) 2022-12-03T01:28:32.850 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834659 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-12-03T01:28:32.850 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834672 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.5.0" (default) 2022-12-03T01:28:32.850 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834681 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default) 2022-12-03T01:28:32.851 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834692 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-12-03T01:28:32.851 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834703 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default) 2022-12-03T01:28:32.852 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834713 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-12-03T01:28:32.852 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834722 I | op-k8sutil: CSI_CLUSTER_NAME="" (default) 2022-12-03T01:28:32.852 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834734 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default) 2022-12-03T01:28:32.853 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834746 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.7.2" 2022-12-03T01:28:32.853 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834840 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:28:32.854 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:28:32.834853 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:28:33.064 INFO:teuthology.orchestra.run.smithi092.stdout:deployment.apps/rook-ceph-tools created 2022-12-03T01:28:33.070 INFO:tasks.rook:Waiting for tools container to start 2022-12-03T01:28:33.071 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:33.195 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:33.196 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 Pending 0 0s 2022-12-03T01:28:33.198 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:33.266 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:33.267 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 1s 2022-12-03T01:28:33.267 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 1s 2022-12-03T01:28:33.268 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 63s 2022-12-03T01:28:33.268 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 0s 2022-12-03T01:28:38.271 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:38.348 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:38.348 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 5s 2022-12-03T01:28:38.350 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:38.426 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:38.427 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 6s 2022-12-03T01:28:38.427 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 6s 2022-12-03T01:28:38.427 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 68s 2022-12-03T01:28:38.428 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 5s 2022-12-03T01:28:43.429 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:43.511 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:43.513 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 10s 2022-12-03T01:28:43.514 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:43.587 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:43.588 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 11s 2022-12-03T01:28:43.588 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 11s 2022-12-03T01:28:43.589 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 73s 2022-12-03T01:28:43.589 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 10s 2022-12-03T01:28:48.591 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:48.661 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:48.662 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 15s 2022-12-03T01:28:48.664 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:48.740 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:48.740 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 16s 2022-12-03T01:28:48.740 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 16s 2022-12-03T01:28:48.741 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 78s 2022-12-03T01:28:48.741 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 15s 2022-12-03T01:28:53.743 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:53.818 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:53.819 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 20s 2022-12-03T01:28:53.821 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:53.893 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:53.893 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 21s 2022-12-03T01:28:53.893 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 21s 2022-12-03T01:28:53.894 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 83s 2022-12-03T01:28:53.894 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 20s 2022-12-03T01:28:58.896 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:28:58.974 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:58.974 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 25s 2022-12-03T01:28:58.976 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:28:59.056 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:28:59.056 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 27s 2022-12-03T01:28:59.057 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 27s 2022-12-03T01:28:59.057 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 89s 2022-12-03T01:28:59.058 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 26s 2022-12-03T01:29:04.061 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:04.141 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:04.142 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 31s 2022-12-03T01:29:04.144 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:04.217 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:04.218 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 32s 2022-12-03T01:29:04.218 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 32s 2022-12-03T01:29:04.218 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 94s 2022-12-03T01:29:04.219 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 31s 2022-12-03T01:29:09.221 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:09.298 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:09.299 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 36s 2022-12-03T01:29:09.300 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:09.372 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:09.373 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 Init:0/1 0 37s 2022-12-03T01:29:09.373 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 Init:0/1 0 37s 2022-12-03T01:29:09.373 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 99s 2022-12-03T01:29:09.373 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 36s 2022-12-03T01:29:14.376 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:14.459 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:14.459 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 41s 2022-12-03T01:29:14.462 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:14.529 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:14.530 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 0/1 PodInitializing 0 42s 2022-12-03T01:29:14.530 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 42s 2022-12-03T01:29:14.530 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 104s 2022-12-03T01:29:14.531 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 41s 2022-12-03T01:29:19.273 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.273858 I | ceph-csi: Detected ceph CSI image version: "v3.7.2" 2022-12-03T01:29:19.286 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284326 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:29:19.286 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284348 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.287 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284360 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:29:19.287 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284366 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.287 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284371 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:29:19.288 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284376 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.288 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284381 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-12-03T01:29:19.288 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284386 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default) 2022-12-03T01:29:19.289 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.284392 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default) 2022-12-03T01:29:19.394 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.394219 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:29:19.394 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.394242 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.394 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.394250 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-12-03T01:29:19.453 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452243 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-12-03T01:29:19.453 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452273 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-12-03T01:29:19.454 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452282 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.454 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452290 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-12-03T01:29:19.455 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452297 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME="" (default) 2022-12-03T01:29:19.455 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.452303 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME_MOUNT="" (default) 2022-12-03T01:29:19.532 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:19.596 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.595009 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-12-03T01:29:19.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.595034 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-12-03T01:29:19.597 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.595043 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-12-03T01:29:19.638 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.638117 I | ceph-csi: successfully started CSI CephFS driver 2022-12-03T01:29:19.639 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.638154 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap) 2022-12-03T01:29:19.642 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:19.643 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 46s 2022-12-03T01:29:19.644 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:19.681 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.676784 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-12-03T01:29:19.681 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.676808 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="File" (configmap) 2022-12-03T01:29:19.765 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.761409 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-12-03T01:29:19.766 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.761435 I | ceph-csi: CSI NFS driver disabled 2022-12-03T01:29:19.766 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.761446 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-12-03T01:29:19.768 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.768439 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-12-03T01:29:19.779 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:19.779 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 0/2 Pending 0 0s 2022-12-03T01:29:19.780 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 Pending 0 0s 2022-12-03T01:29:19.780 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 Pending 0 0s 2022-12-03T01:29:19.781 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 0s 2022-12-03T01:29:19.781 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 0/2 ContainerCreating 0 0s 2022-12-03T01:29:19.781 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 Pending 0 0s 2022-12-03T01:29:19.782 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 Pending 0 0s 2022-12-03T01:29:19.782 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-csi-detect-version-7mvtf 1/1 Terminating 0 47s 2022-12-03T01:29:19.783 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 47s 2022-12-03T01:29:19.783 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 109s 2022-12-03T01:29:19.783 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 46s 2022-12-03T01:29:19.932 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:19.932554 I | ceph-csi: successfully removed CSI NFS driver 2022-12-03T01:29:24.785 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:24.855 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:24.856 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 51s 2022-12-03T01:29:24.858 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:24.966 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:24.966 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 0/2 ContainerCreating 0 5s 2022-12-03T01:29:24.966 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 5s 2022-12-03T01:29:24.966 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 5s 2022-12-03T01:29:24.967 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 5s 2022-12-03T01:29:24.967 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 5s 2022-12-03T01:29:24.967 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 0/2 ContainerCreating 0 5s 2022-12-03T01:29:24.967 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 5s 2022-12-03T01:29:24.968 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 5s 2022-12-03T01:29:24.968 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 52s 2022-12-03T01:29:24.968 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 114s 2022-12-03T01:29:24.968 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 51s 2022-12-03T01:29:29.971 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:30.058 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:30.058 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 57s 2022-12-03T01:29:30.060 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:30.144 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:30.145 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 0/2 ContainerCreating 0 11s 2022-12-03T01:29:30.145 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 11s 2022-12-03T01:29:30.145 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 11s 2022-12-03T01:29:30.146 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 11s 2022-12-03T01:29:30.146 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 11s 2022-12-03T01:29:30.146 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 0/2 ContainerCreating 0 11s 2022-12-03T01:29:30.147 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 11s 2022-12-03T01:29:30.147 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 11s 2022-12-03T01:29:30.147 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 58s 2022-12-03T01:29:30.147 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m 2022-12-03T01:29:30.148 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 57s 2022-12-03T01:29:35.150 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:35.231 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:35.231 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 62s 2022-12-03T01:29:35.233 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:35.330 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:35.331 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 0/2 ContainerCreating 0 16s 2022-12-03T01:29:35.331 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 16s 2022-12-03T01:29:35.332 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 16s 2022-12-03T01:29:35.332 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 16s 2022-12-03T01:29:35.333 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 16s 2022-12-03T01:29:35.333 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 0/2 ContainerCreating 0 16s 2022-12-03T01:29:35.333 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 16s 2022-12-03T01:29:35.334 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 16s 2022-12-03T01:29:35.334 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 63s 2022-12-03T01:29:35.335 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m5s 2022-12-03T01:29:35.336 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 62s 2022-12-03T01:29:40.337 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:40.425 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:40.425 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 67s 2022-12-03T01:29:40.427 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:40.511 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:40.512 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 0/2 ContainerCreating 0 21s 2022-12-03T01:29:40.512 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 21s 2022-12-03T01:29:40.512 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 21s 2022-12-03T01:29:40.513 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 21s 2022-12-03T01:29:40.513 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 21s 2022-12-03T01:29:40.513 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 21s 2022-12-03T01:29:40.513 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 21s 2022-12-03T01:29:40.514 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 21s 2022-12-03T01:29:40.514 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 68s 2022-12-03T01:29:40.514 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m10s 2022-12-03T01:29:40.515 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 67s 2022-12-03T01:29:45.517 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:45.595 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:45.596 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 72s 2022-12-03T01:29:45.598 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:45.693 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:45.694 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 26s 2022-12-03T01:29:45.694 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 26s 2022-12-03T01:29:45.694 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 26s 2022-12-03T01:29:45.694 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 26s 2022-12-03T01:29:45.695 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 0/2 ContainerCreating 0 26s 2022-12-03T01:29:45.695 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 26s 2022-12-03T01:29:45.695 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 26s 2022-12-03T01:29:45.696 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 26s 2022-12-03T01:29:45.696 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-detect-version-q65tw 0/1 PodInitializing 0 73s 2022-12-03T01:29:45.696 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m15s 2022-12-03T01:29:45.696 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 72s 2022-12-03T01:29:46.562 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.562868 I | ceph-spec: detected ceph image version: "18.0.0-872 reef" 2022-12-03T01:29:46.562 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.562893 I | ceph-cluster-controller: validating ceph version from provided image 2022-12-03T01:29:46.563 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.562906 W | ceph-cluster-controller: unsupported ceph version detected: "18.0.0-872 reef", pursuing 2022-12-03T01:29:46.586 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.586312 I | ceph-cluster-controller: cluster "rook-ceph": version "18.0.0-872 reef" detected for image "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:48d203d9b836cc224eb4085f6198d948c6fa1bb1" 2022-12-03T01:29:46.704 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.704457 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-12-03 01:29:46.679351423 +0000 UTC m=+85.765551394 LastTransitionTime:2022-12-03 01:29:46.679351354 +0000 UTC m=+85.765551328}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:29:46.767 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.767356 I | op-mon: start running mons 2022-12-03T01:29:46.825 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.825842 I | ceph-spec: creating mon secrets for a new cluster 2022-12-03T01:29:46.846 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.847328 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-12-03T01:29:46.853 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:46.854441 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[],"namespace":""}] data: mapping:{"node":{}} maxMonId:-1 outOfQuorum:] 2022-12-03T01:29:47.560 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:47.561035 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:29:47.561 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:47.561161 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:29:49.126 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.126981 I | op-mon: targeting the mon count 3 2022-12-03T01:29:49.180 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.180662 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-12-03T01:29:49.297 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.298216 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-12-03T01:29:49.504 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.504855 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-12-03T01:29:49.531 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.532214 I | op-mon: waiting for canary pod creation rook-ceph-mon-b-canary 2022-12-03T01:29:49.743 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.743770 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi121 2022-12-03T01:29:49.743 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:49.743795 I | op-mon: mon a assigned to node smithi121 2022-12-03T01:29:50.108 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:50.108773 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi138 2022-12-03T01:29:50.108 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:50.108799 I | op-mon: mon c assigned to node smithi138 2022-12-03T01:29:50.698 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:50.770 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:50.770 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 77s 2022-12-03T01:29:50.772 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:50.861 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:50.862 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 31s 2022-12-03T01:29:50.862 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 31s 2022-12-03T01:29:50.862 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 0/5 ContainerCreating 0 31s 2022-12-03T01:29:50.863 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 0/2 ContainerCreating 0 31s 2022-12-03T01:29:50.863 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 31s 2022-12-03T01:29:50.863 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 31s 2022-12-03T01:29:50.863 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 31s 2022-12-03T01:29:50.863 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 0/5 ContainerCreating 0 31s 2022-12-03T01:29:50.864 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 0/1 ContainerCreating 0 1s 2022-12-03T01:29:50.864 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 0/1 ContainerCreating 0 1s 2022-12-03T01:29:50.864 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 0/1 ContainerCreating 0 1s 2022-12-03T01:29:50.864 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m20s 2022-12-03T01:29:50.865 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 77s 2022-12-03T01:29:52.787 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:52.787050 I | ceph-spec: parsing mon endpoints: 2022-12-03T01:29:52.788 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:52.787075 W | ceph-spec: ignoring invalid monitor 2022-12-03T01:29:52.788 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:52.787112 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-12-03T01:29:52.788 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:52.787122 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-12-03T01:29:52.788 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:52.787682 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-12-03T01:29:52.788 INFO:tasks.rook.operator.smithi092.stdout:I1203 01:29:52.787816 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-12-03T01:29:54.565 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.565237 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi138 2022-12-03T01:29:54.565 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.565260 I | op-mon: mon b assigned to node smithi138 2022-12-03T01:29:54.571 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.571681 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-12-03T01:29:54.601 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.602478 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-12-03T01:29:54.627 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.627853 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-12-03T01:29:54.710 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.710966 I | op-mon: creating mon a 2022-12-03T01:29:54.879 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:54.879638 I | op-mon: mon "a" ip is 10.97.213.111 2022-12-03T01:29:55.060 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.061230 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-12-03T01:29:55.061 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.061356 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-12-03T01:29:55.061 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.061729 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.213.111:6789"],"namespace":""}] data:a=10.97.213.111:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:-1 outOfQuorum:] 2022-12-03T01:29:55.112 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.112315 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:29:55.113 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.112560 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:29:55.174 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.174910 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-12-03T01:29:55.240 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.241266 I | op-mon: updating maxMonID from -1 to 0 2022-12-03T01:29:55.736 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.736718 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.213.111:6789"],"namespace":""}] data:a=10.97.213.111:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:0 outOfQuorum:] 2022-12-03T01:29:55.736 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.736750 I | op-mon: waiting for mon quorum with [a] 2022-12-03T01:29:55.866 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:29:55.940 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:29:55.941458 I | op-mon: mons running: [a] 2022-12-03T01:29:55.948 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:55.948 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 82s 2022-12-03T01:29:55.950 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:29:56.035 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:29:56.035 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 37s 2022-12-03T01:29:56.035 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 37s 2022-12-03T01:29:56.036 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 37s 2022-12-03T01:29:56.036 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 37s 2022-12-03T01:29:56.036 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 37s 2022-12-03T01:29:56.036 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 37s 2022-12-03T01:29:56.037 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 37s 2022-12-03T01:29:56.037 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 37s 2022-12-03T01:29:56.037 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 1s 2022-12-03T01:29:56.038 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 7s 2022-12-03T01:29:56.038 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 7s 2022-12-03T01:29:56.038 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 7s 2022-12-03T01:29:56.038 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m26s 2022-12-03T01:29:56.039 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 83s 2022-12-03T01:30:01.041 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:01.118 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:01.118 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 88s 2022-12-03T01:30:01.119 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:01.208 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:01.208 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 42s 2022-12-03T01:30:01.209 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 42s 2022-12-03T01:30:01.209 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 42s 2022-12-03T01:30:01.209 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 42s 2022-12-03T01:30:01.209 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 42s 2022-12-03T01:30:01.209 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 42s 2022-12-03T01:30:01.210 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 42s 2022-12-03T01:30:01.210 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 42s 2022-12-03T01:30:01.210 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 6s 2022-12-03T01:30:01.210 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 12s 2022-12-03T01:30:01.211 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 12s 2022-12-03T01:30:01.211 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 12s 2022-12-03T01:30:01.211 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m31s 2022-12-03T01:30:01.211 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 88s 2022-12-03T01:30:06.214 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:06.286 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:06.287 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 93s 2022-12-03T01:30:06.290 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:06.373 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:06.374 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 47s 2022-12-03T01:30:06.374 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 0/5 ContainerCreating 0 47s 2022-12-03T01:30:06.374 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 47s 2022-12-03T01:30:06.375 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 47s 2022-12-03T01:30:06.375 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 47s 2022-12-03T01:30:06.376 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 47s 2022-12-03T01:30:06.376 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 0/5 ContainerCreating 0 47s 2022-12-03T01:30:06.376 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 47s 2022-12-03T01:30:06.377 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 11s 2022-12-03T01:30:06.377 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 17s 2022-12-03T01:30:06.378 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 17s 2022-12-03T01:30:06.378 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 17s 2022-12-03T01:30:06.378 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m36s 2022-12-03T01:30:06.379 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 93s 2022-12-03T01:30:11.381 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:11.459 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:11.459 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 98s 2022-12-03T01:30:11.462 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:11.545 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:11.546 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 52s 2022-12-03T01:30:11.546 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 52s 2022-12-03T01:30:11.547 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 52s 2022-12-03T01:30:11.547 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 52s 2022-12-03T01:30:11.547 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 52s 2022-12-03T01:30:11.547 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 52s 2022-12-03T01:30:11.548 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 52s 2022-12-03T01:30:11.548 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 52s 2022-12-03T01:30:11.548 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 16s 2022-12-03T01:30:11.548 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 22s 2022-12-03T01:30:11.548 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 22s 2022-12-03T01:30:11.549 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 22s 2022-12-03T01:30:11.549 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m41s 2022-12-03T01:30:11.549 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 98s 2022-12-03T01:30:16.075 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:16.074717 I | op-mon: mons running: [a] 2022-12-03T01:30:16.551 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:16.626 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:16.627 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 103s 2022-12-03T01:30:16.628 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:16.736 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:16.736 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 57s 2022-12-03T01:30:16.737 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 57s 2022-12-03T01:30:16.737 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 57s 2022-12-03T01:30:16.738 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 57s 2022-12-03T01:30:16.738 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 57s 2022-12-03T01:30:16.739 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 57s 2022-12-03T01:30:16.739 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 57s 2022-12-03T01:30:16.740 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 57s 2022-12-03T01:30:16.740 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 21s 2022-12-03T01:30:16.740 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 27s 2022-12-03T01:30:16.740 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 27s 2022-12-03T01:30:16.741 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 27s 2022-12-03T01:30:16.741 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m46s 2022-12-03T01:30:16.741 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 103s 2022-12-03T01:30:21.743 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:21.821 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:21.822 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 108s 2022-12-03T01:30:21.832 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:21.928 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:21.929 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 62s 2022-12-03T01:30:21.929 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 62s 2022-12-03T01:30:21.929 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 62s 2022-12-03T01:30:21.930 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 62s 2022-12-03T01:30:21.930 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 62s 2022-12-03T01:30:21.930 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 62s 2022-12-03T01:30:21.930 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 62s 2022-12-03T01:30:21.931 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 62s 2022-12-03T01:30:21.931 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Init:0/2 0 26s 2022-12-03T01:30:21.931 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-canary-66965ffdc4-ptz52 1/1 Terminating 0 32s 2022-12-03T01:30:21.931 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-canary-5db77ddc9b-h6rm7 1/1 Terminating 0 32s 2022-12-03T01:30:21.932 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 32s 2022-12-03T01:30:21.932 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m51s 2022-12-03T01:30:21.932 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 108s 2022-12-03T01:30:26.934 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:27.068 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:27.069 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 114s 2022-12-03T01:30:27.070 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:27.154 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:27.154 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 68s 2022-12-03T01:30:27.155 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 68s 2022-12-03T01:30:27.155 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 68s 2022-12-03T01:30:27.155 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 68s 2022-12-03T01:30:27.155 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 68s 2022-12-03T01:30:27.156 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 68s 2022-12-03T01:30:27.156 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 68s 2022-12-03T01:30:27.156 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 68s 2022-12-03T01:30:27.156 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Running 0 32s 2022-12-03T01:30:27.157 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-canary-7476bc4b54-7kjrx 1/1 Terminating 0 38s 2022-12-03T01:30:27.157 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 2m57s 2022-12-03T01:30:27.157 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 114s 2022-12-03T01:30:32.160 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:32.241 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:32.242 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 119s 2022-12-03T01:30:32.244 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:32.322 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:32.323 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 73s 2022-12-03T01:30:32.323 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 73s 2022-12-03T01:30:32.323 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 73s 2022-12-03T01:30:32.323 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 73s 2022-12-03T01:30:32.324 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 73s 2022-12-03T01:30:32.324 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 73s 2022-12-03T01:30:32.324 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 73s 2022-12-03T01:30:32.324 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 73s 2022-12-03T01:30:32.325 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 0/1 Running 0 37s 2022-12-03T01:30:32.325 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 3m2s 2022-12-03T01:30:32.325 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 119s 2022-12-03T01:30:36.203 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.203675 I | op-mon: mons running: [a] 2022-12-03T01:30:36.426 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.427101 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:30:36.426 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.427124 I | op-mon: mons created: 1 2022-12-03T01:30:36.648 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.649193 I | op-mon: waiting for mon quorum with [a] 2022-12-03T01:30:36.655 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.656261 I | op-mon: mons running: [a] 2022-12-03T01:30:36.881 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.881134 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:30:36.882 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:36.881444 I | op-config: applying ceph settings: 2022-12-03T01:30:36.882 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:30:36.882 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool delete = true 2022-12-03T01:30:36.882 INFO:tasks.rook.operator.smithi092.stdout:mon cluster log file = 2022-12-03T01:30:36.882 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool size one = true 2022-12-03T01:30:37.211 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.211927 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:30:37.211 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.212234 I | op-config: applying ceph settings: 2022-12-03T01:30:37.211 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:30:37.212 INFO:tasks.rook.operator.smithi092.stdout:log to file = false 2022-12-03T01:30:37.326 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:37.401 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:37.401 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m4s 2022-12-03T01:30:37.404 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:37.470 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.470795 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:30:37.470 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.470863 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:30:37.495 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:37.496 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 78s 2022-12-03T01:30:37.496 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 78s 2022-12-03T01:30:37.496 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 78s 2022-12-03T01:30:37.498 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 78s 2022-12-03T01:30:37.498 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 78s 2022-12-03T01:30:37.498 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 78s 2022-12-03T01:30:37.499 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 78s 2022-12-03T01:30:37.499 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 78s 2022-12-03T01:30:37.499 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Running 0 42s 2022-12-03T01:30:37.500 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 3m7s 2022-12-03T01:30:37.500 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m4s 2022-12-03T01:30:37.692 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.693059 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:30:37.692 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.693090 I | op-mon: creating mon b 2022-12-03T01:30:37.877 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.877205 I | op-mon: mon "a" ip is 10.97.213.111 2022-12-03T01:30:37.917 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:37.918380 I | op-mon: mon "b" ip is 10.98.170.74 2022-12-03T01:30:38.016 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.017434 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.213.111:6789","10.98.170.74:6789"],"namespace":""}] data:b=10.98.170.74:6789,a=10.97.213.111:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:0 outOfQuorum:] 2022-12-03T01:30:38.036 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.037639 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:30:38.037 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.037923 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:30:38.313 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.314927 I | op-mon: 1 of 2 expected mon deployments exist. creating new deployment(s). 2022-12-03T01:30:38.320 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.321543 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-12-03T01:30:38.334 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.335149 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-12-03T01:30:38.499 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:38.500458 I | op-mon: updating maxMonID from 0 to 1 2022-12-03T01:30:39.321 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.322163 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.170.74:6789","10.97.213.111:6789"],"namespace":""}] data:a=10.97.213.111:6789,b=10.98.170.74:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:1 outOfQuorum:] 2022-12-03T01:30:39.321 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.322193 I | op-mon: waiting for mon quorum with [a b] 2022-12-03T01:30:39.700 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.701224 I | op-mon: mon b is not yet running 2022-12-03T01:30:39.702 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.701248 I | op-mon: mons running: [a] 2022-12-03T01:30:39.924 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.924686 I | op-mon: Monitors in quorum: [a] 2022-12-03T01:30:39.925 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:39.924711 I | op-mon: mons created: 2 2022-12-03T01:30:40.192 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:40.193994 I | op-mon: waiting for mon quorum with [a b] 2022-12-03T01:30:40.205 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:40.206334 I | op-mon: mon b is not yet running 2022-12-03T01:30:40.205 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:40.206355 I | op-mon: mons running: [a] 2022-12-03T01:30:42.503 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:42.583 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:42.584 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m9s 2022-12-03T01:30:42.587 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:42.662 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:42.662 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 83s 2022-12-03T01:30:42.663 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 83s 2022-12-03T01:30:42.663 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 83s 2022-12-03T01:30:42.664 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 83s 2022-12-03T01:30:42.664 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 83s 2022-12-03T01:30:42.664 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 83s 2022-12-03T01:30:42.664 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 83s 2022-12-03T01:30:42.665 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 83s 2022-12-03T01:30:42.665 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Running 0 47s 2022-12-03T01:30:42.665 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-6668c88b8d-2h5f2 0/1 Init:1/2 0 4s 2022-12-03T01:30:42.665 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 3m12s 2022-12-03T01:30:42.666 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m9s 2022-12-03T01:30:45.229 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:45.230254 I | op-mon: mons running: [a b] 2022-12-03T01:30:47.666 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:47.741 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:47.742 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m14s 2022-12-03T01:30:47.744 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:47.840 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:47.840 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 88s 2022-12-03T01:30:47.841 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 88s 2022-12-03T01:30:47.841 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 88s 2022-12-03T01:30:47.841 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 88s 2022-12-03T01:30:47.841 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 88s 2022-12-03T01:30:47.842 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 88s 2022-12-03T01:30:47.842 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 88s 2022-12-03T01:30:47.842 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 88s 2022-12-03T01:30:47.842 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Running 0 52s 2022-12-03T01:30:47.843 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-6668c88b8d-2h5f2 0/1 Running 0 9s 2022-12-03T01:30:47.843 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 3m17s 2022-12-03T01:30:47.843 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 0/1 ContainerCreating 0 2m14s 2022-12-03T01:30:51.498 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.499992 I | op-mon: Monitors in quorum: [a b] 2022-12-03T01:30:51.500 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.500375 I | op-config: applying ceph settings: 2022-12-03T01:30:51.501 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:30:51.501 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool delete = true 2022-12-03T01:30:51.501 INFO:tasks.rook.operator.smithi092.stdout:mon cluster log file = 2022-12-03T01:30:51.501 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool size one = true 2022-12-03T01:30:51.719 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.720645 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:30:51.720 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.721084 I | op-config: applying ceph settings: 2022-12-03T01:30:51.720 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:30:51.720 INFO:tasks.rook.operator.smithi092.stdout:log to file = false 2022-12-03T01:30:51.939 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.939632 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:30:51.939 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:51.939730 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:30:52.162 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.163736 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:30:52.163 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.163763 I | op-mon: creating mon c 2022-12-03T01:30:52.259 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.260980 I | op-mon: mon "a" ip is 10.97.213.111 2022-12-03T01:30:52.362 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.362779 I | op-mon: mon "b" ip is 10.98.170.74 2022-12-03T01:30:52.401 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.397608 I | op-mon: mon "c" ip is 10.111.191.134 2022-12-03T01:30:52.575 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:52.576824 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.213.111:6789","10.98.170.74:6789","10.111.191.134:6789"],"namespace":""}] data:b=10.98.170.74:6789,c=10.111.191.134:6789,a=10.97.213.111:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:1 outOfQuorum:] 2022-12-03T01:30:52.845 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-12-03T01:30:52.919 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:52.920 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Running 0 2m19s 2022-12-03T01:30:52.922 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:30:53.008 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:30:53.008 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 93s 2022-12-03T01:30:53.008 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 93s 2022-12-03T01:30:53.009 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 93s 2022-12-03T01:30:53.009 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 93s 2022-12-03T01:30:53.009 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 93s 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 93s 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 93s 2022-12-03T01:30:53.010 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 93s 2022-12-03T01:30:53.011 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Running 0 57s 2022-12-03T01:30:53.011 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-6668c88b8d-2h5f2 0/1 Running 0 14s 2022-12-03T01:30:53.012 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 3m22s 2022-12-03T01:30:53.012 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Running 0 2m19s 2022-12-03T01:30:53.013 INFO:tasks.rook:Waiting for 12 OSDs 2022-12-03T01:30:53.013 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:30:53.166 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.167733 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-12-03T01:30:53.167 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.168046 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-12-03T01:30:53.397 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:30:53.419 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:30:53.420 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:30:53.598 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.599341 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-12-03T01:30:53.640 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.640594 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-12-03T01:30:53.649 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.650641 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-12-03T01:30:53.654 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.655638 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-12-03T01:30:53.661 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.663508 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-12-03T01:30:53.767 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:53.766696 I | op-mon: updating maxMonID from 1 to 2 2022-12-03T01:30:54.575 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:54.576941 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.213.111:6789","10.98.170.74:6789","10.111.191.134:6789"],"namespace":""}] data:a=10.97.213.111:6789,b=10.98.170.74:6789,c=10.111.191.134:6789 mapping:{"node":{"a":{"Name":"smithi121","Hostname":"smithi121","Address":"172.21.15.121"},"b":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"},"c":{"Name":"smithi138","Hostname":"smithi138","Address":"172.21.15.138"}}} maxMonId:2 outOfQuorum:] 2022-12-03T01:30:54.576 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:54.576967 I | op-mon: waiting for mon quorum with [a b c] 2022-12-03T01:30:55.170 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.170822 I | op-mon: mon c is not yet running 2022-12-03T01:30:55.170 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.170850 I | op-mon: mons running: [a b] 2022-12-03T01:30:55.399 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.400212 I | op-mon: Monitors in quorum: [a b] 2022-12-03T01:30:55.399 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.400239 I | op-mon: mons created: 3 2022-12-03T01:30:55.617 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.618203 I | op-mon: waiting for mon quorum with [a b c] 2022-12-03T01:30:55.771 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.772508 I | op-mon: mon c is not yet running 2022-12-03T01:30:55.771 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:30:55.772535 I | op-mon: mons running: [a b] 2022-12-03T01:31:00.792 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:00.793927 I | op-mon: mons running: [a b c] 2022-12-03T01:31:03.420 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:05.429 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.430802 I | op-mon: Monitors in quorum: [a b c] 2022-12-03T01:31:05.430 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.431123 I | op-config: applying ceph settings: 2022-12-03T01:31:05.430 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:31:05.431 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool delete = true 2022-12-03T01:31:05.431 INFO:tasks.rook.operator.smithi092.stdout:mon cluster log file = 2022-12-03T01:31:05.431 INFO:tasks.rook.operator.smithi092.stdout:mon allow pool size one = true 2022-12-03T01:31:05.500 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:05.520 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:31:05.520 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:31:05.685 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.685770 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:31:05.685 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.686313 I | op-config: applying ceph settings: 2022-12-03T01:31:05.686 INFO:tasks.rook.operator.smithi092.stdout:[global] 2022-12-03T01:31:05.686 INFO:tasks.rook.operator.smithi092.stdout:log to file = false 2022-12-03T01:31:05.904 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.906186 I | op-config: successfully applied settings to the mon configuration database 2022-12-03T01:31:05.905 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:05.906242 I | op-config: deleting "log file" option from the mon configuration database 2022-12-03T01:31:06.122 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:06.123917 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-12-03T01:31:06.123 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:06.123946 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-12-03T01:31:06.476 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:06.477541 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-12-03T01:31:06.939 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:06.940916 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-12-03T01:31:07.272 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:07.274416 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-12-03T01:31:07.665 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:07.666267 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-12-03T01:31:07.666 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:07.666290 I | cephclient: getting or creating ceph auth key "client.crash" 2022-12-03T01:31:08.065 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.066481 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-12-03T01:31:08.065 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.066533 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-12-03T01:31:08.065 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.066544 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-12-03T01:31:08.379 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.380423 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-12-03T01:31:08.461 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.462903 I | op-mgr: start running mgr 2022-12-03T01:31:08.461 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.462933 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-12-03T01:31:08.881 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:08.883258 I | op-config: setting "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-12-03T01:31:09.208 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:09.210182 I | op-config: successfully set "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-12-03T01:31:09.209 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:09.210206 I | op-config: insecure global ID is now disabled 2022-12-03T01:31:09.504 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:09.506072 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi121": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:31:15.522 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:15.869 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:15.885 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:31:15.886 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:31:25.887 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:26.426 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:31:26.426 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:31:30.356 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.358447 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-12-03T01:31:30.360 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.362157 I | op-mgr: setting services to point to mgr "a" 2022-12-03T01:31:30.428 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.429977 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-12-03T01:31:30.428 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.430099 I | op-mgr: successful modules: balancer 2022-12-03T01:31:30.467 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.469287 I | op-osd: start running osds in namespace "rook-ceph" 2022-12-03T01:31:30.467 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.469389 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-12-03T01:31:30.474 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.476470 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-12-03T01:31:30.476 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.478460 I | op-osd: creating 12 new PVCs for device set "scratch" 2022-12-03T01:31:30.545 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.547720 I | op-osd: successfully provisioned PVC "scratch-data-0nxlcm" 2022-12-03T01:31:30.582 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.584282 I | op-osd: successfully provisioned PVC "scratch-data-1gl9wr" 2022-12-03T01:31:30.605 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.607667 I | op-osd: successfully provisioned PVC "scratch-data-2hkmfs" 2022-12-03T01:31:30.640 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.642295 I | op-osd: successfully provisioned PVC "scratch-data-3hkxqr" 2022-12-03T01:31:30.670 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.672374 I | op-osd: successfully provisioned PVC "scratch-data-45wsd7" 2022-12-03T01:31:30.707 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.708903 I | op-osd: successfully provisioned PVC "scratch-data-5d42s2" 2022-12-03T01:31:30.722 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.724189 I | op-osd: successfully provisioned PVC "scratch-data-6vbd4r" 2022-12-03T01:31:30.811 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.813803 I | op-osd: successfully provisioned PVC "scratch-data-7zz4lb" 2022-12-03T01:31:30.972 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:30.974648 I | op-osd: successfully provisioned PVC "scratch-data-8zz8z7" 2022-12-03T01:31:31.191 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.192827 I | op-osd: successfully provisioned PVC "scratch-data-9r4fv9" 2022-12-03T01:31:31.372 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.374325 I | op-osd: successfully provisioned PVC "scratch-data-10vzmld" 2022-12-03T01:31:31.502 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.504432 I | op-mgr: successful modules: dashboard 2022-12-03T01:31:31.503 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.505262 I | op-mgr: successful modules: mgr module(s) from the spec 2022-12-03T01:31:31.531 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.533654 I | op-mgr: successful modules: prometheus 2022-12-03T01:31:31.599 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.601470 I | op-osd: successfully provisioned PVC "scratch-data-11m4fvt" 2022-12-03T01:31:31.603 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:31.605536 I | op-osd: OSD will have its main bluestore block on "scratch-data-0nxlcm" 2022-12-03T01:31:32.051 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.053129 I | op-osd: started OSD provisioning job for PVC "scratch-data-0nxlcm" 2022-12-03T01:31:32.052 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.053158 I | op-osd: OSD will have its main bluestore block on "scratch-data-1gl9wr" 2022-12-03T01:31:32.447 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.449332 I | op-osd: started OSD provisioning job for PVC "scratch-data-1gl9wr" 2022-12-03T01:31:32.448 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.449391 I | op-osd: OSD will have its main bluestore block on "scratch-data-2hkmfs" 2022-12-03T01:31:32.834 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.836509 I | op-osd: started OSD provisioning job for PVC "scratch-data-2hkmfs" 2022-12-03T01:31:32.835 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:32.836531 I | op-osd: OSD will have its main bluestore block on "scratch-data-3hkxqr" 2022-12-03T01:31:33.294 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:33.295677 I | op-osd: started OSD provisioning job for PVC "scratch-data-3hkxqr" 2022-12-03T01:31:33.294 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:33.295702 I | op-osd: OSD will have its main bluestore block on "scratch-data-45wsd7" 2022-12-03T01:31:33.680 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:33.682319 I | op-osd: started OSD provisioning job for PVC "scratch-data-45wsd7" 2022-12-03T01:31:33.680 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:33.682343 I | op-osd: OSD will have its main bluestore block on "scratch-data-5d42s2" 2022-12-03T01:31:34.054 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.056492 I | op-osd: started OSD provisioning job for PVC "scratch-data-5d42s2" 2022-12-03T01:31:34.055 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.056513 I | op-osd: OSD will have its main bluestore block on "scratch-data-6vbd4r" 2022-12-03T01:31:34.478 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.480745 I | op-osd: started OSD provisioning job for PVC "scratch-data-6vbd4r" 2022-12-03T01:31:34.479 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.480759 I | op-osd: OSD will have its main bluestore block on "scratch-data-7zz4lb" 2022-12-03T01:31:34.889 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.891255 I | op-osd: started OSD provisioning job for PVC "scratch-data-7zz4lb" 2022-12-03T01:31:34.889 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:34.891293 I | op-osd: OSD will have its main bluestore block on "scratch-data-8zz8z7" 2022-12-03T01:31:35.214 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:35.215911 I | op-osd: started OSD provisioning job for PVC "scratch-data-8zz8z7" 2022-12-03T01:31:35.214 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:35.215927 I | op-osd: OSD will have its main bluestore block on "scratch-data-9r4fv9" 2022-12-03T01:31:35.661 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:35.663903 I | op-osd: started OSD provisioning job for PVC "scratch-data-9r4fv9" 2022-12-03T01:31:35.662 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:35.663923 I | op-osd: OSD will have its main bluestore block on "scratch-data-10vzmld" 2022-12-03T01:31:36.078 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.080323 I | op-osd: started OSD provisioning job for PVC "scratch-data-10vzmld" 2022-12-03T01:31:36.078 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.080336 I | op-osd: OSD will have its main bluestore block on "scratch-data-11m4fvt" 2022-12-03T01:31:36.399 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.401459 I | op-osd: started OSD provisioning job for PVC "scratch-data-11m4fvt" 2022-12-03T01:31:36.400 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.401476 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-12-03T01:31:36.400 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.401479 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-12-03T01:31:36.428 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:36.564 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566044 I | op-osd: OSD orchestration status for PVC scratch-data-0nxlcm is "starting" 2022-12-03T01:31:36.565 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566061 I | op-osd: OSD orchestration status for PVC scratch-data-10vzmld is "starting" 2022-12-03T01:31:36.565 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566066 I | op-osd: OSD orchestration status for PVC scratch-data-11m4fvt is "starting" 2022-12-03T01:31:36.565 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566070 I | op-osd: OSD orchestration status for PVC scratch-data-1gl9wr is "starting" 2022-12-03T01:31:36.566 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566074 I | op-osd: OSD orchestration status for PVC scratch-data-2hkmfs is "starting" 2022-12-03T01:31:36.566 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566078 I | op-osd: OSD orchestration status for PVC scratch-data-3hkxqr is "starting" 2022-12-03T01:31:36.566 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566082 I | op-osd: OSD orchestration status for PVC scratch-data-45wsd7 is "starting" 2022-12-03T01:31:36.567 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566085 I | op-osd: OSD orchestration status for PVC scratch-data-5d42s2 is "starting" 2022-12-03T01:31:36.567 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566089 I | op-osd: OSD orchestration status for PVC scratch-data-6vbd4r is "starting" 2022-12-03T01:31:36.567 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566094 I | op-osd: OSD orchestration status for PVC scratch-data-7zz4lb is "starting" 2022-12-03T01:31:36.567 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566098 I | op-osd: OSD orchestration status for PVC scratch-data-8zz8z7 is "starting" 2022-12-03T01:31:36.568 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:36.566101 I | op-osd: OSD orchestration status for PVC scratch-data-9r4fv9 is "starting" 2022-12-03T01:31:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:36.992 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-12-03T01:31:36.992 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:31:42.705 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:42.707119 I | op-mgr: successful modules: orchestrator modules 2022-12-03T01:31:43.749 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:43.751316 I | op-osd: OSD orchestration status for node scratch-data-45wsd7 is "orchestrating" 2022-12-03T01:31:44.117 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:44.119526 I | op-osd: OSD orchestration status for PVC scratch-data-45wsd7 is "orchestrating" 2022-12-03T01:31:44.469 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:44.471041 I | op-osd: OSD orchestration status for node scratch-data-0nxlcm is "orchestrating" 2022-12-03T01:31:44.835 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:44.826255 I | op-osd: OSD orchestration status for PVC scratch-data-0nxlcm is "orchestrating" 2022-12-03T01:31:45.932 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:45.934472 I | op-osd: OSD orchestration status for node scratch-data-1gl9wr is "orchestrating" 2022-12-03T01:31:46.471 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:46.473242 I | op-osd: OSD orchestration status for PVC scratch-data-1gl9wr is "orchestrating" 2022-12-03T01:31:46.949 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:46.951110 I | op-osd: OSD orchestration status for node scratch-data-5d42s2 is "orchestrating" 2022-12-03T01:31:46.993 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:47.096 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:47.098443 I | op-osd: OSD orchestration status for node scratch-data-2hkmfs is "orchestrating" 2022-12-03T01:31:47.407 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:47.409652 I | op-osd: OSD orchestration status for PVC scratch-data-5d42s2 is "orchestrating" 2022-12-03T01:31:47.466 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:47.468035 I | op-osd: OSD orchestration status for PVC scratch-data-2hkmfs is "orchestrating" 2022-12-03T01:31:47.626 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:47.648 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":3,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1670031106,"num_remapped_pgs":0} 2022-12-03T01:31:47.649 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:31:48.799 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:48.801814 I | op-osd: OSD orchestration status for node scratch-data-3hkxqr is "orchestrating" 2022-12-03T01:31:49.199 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:49.201335 I | op-osd: OSD orchestration status for PVC scratch-data-3hkxqr is "orchestrating" 2022-12-03T01:31:49.765 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:49.767989 I | op-osd: OSD orchestration status for node scratch-data-6vbd4r is "orchestrating" 2022-12-03T01:31:49.774 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:49.776367 I | op-osd: OSD orchestration status for node scratch-data-7zz4lb is "orchestrating" 2022-12-03T01:31:50.141 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:50.143123 I | op-osd: OSD orchestration status for PVC scratch-data-7zz4lb is "orchestrating" 2022-12-03T01:31:50.164 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:50.166998 I | op-osd: OSD orchestration status for PVC scratch-data-6vbd4r is "orchestrating" 2022-12-03T01:31:52.232 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.234749 I | op-osd: OSD orchestration status for PVC scratch-data-0nxlcm is "completed" 2022-12-03T01:31:52.233 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.234829 I | op-osd: creating OSD 0 on PVC "scratch-data-0nxlcm" 2022-12-03T01:31:52.233 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.234846 I | op-osd: OSD will have its main bluestore block on "scratch-data-0nxlcm" 2022-12-03T01:31:52.667 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.668910 I | op-osd: OSD orchestration status for PVC scratch-data-45wsd7 is "completed" 2022-12-03T01:31:52.667 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.668933 I | op-osd: creating OSD 1 on PVC "scratch-data-45wsd7" 2022-12-03T01:31:52.667 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.668943 I | op-osd: OSD will have its main bluestore block on "scratch-data-45wsd7" 2022-12-03T01:31:52.923 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:52.925693 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi121": the object has been modified; please apply your changes to the latest version and try again 2022-12-03T01:31:54.268 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.270374 I | op-osd: OSD orchestration status for PVC scratch-data-1gl9wr is "completed" 2022-12-03T01:31:54.268 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.270407 I | op-osd: creating OSD 2 on PVC "scratch-data-1gl9wr" 2022-12-03T01:31:54.269 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.270422 I | op-osd: OSD will have its main bluestore block on "scratch-data-1gl9wr" 2022-12-03T01:31:54.451 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.453730 I | op-osd: OSD orchestration status for PVC scratch-data-2hkmfs is "completed" 2022-12-03T01:31:54.452 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.453759 I | op-osd: creating OSD 3 on PVC "scratch-data-2hkmfs" 2022-12-03T01:31:54.452 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:54.453769 I | op-osd: OSD will have its main bluestore block on "scratch-data-2hkmfs" 2022-12-03T01:31:55.773 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:55.775340 I | op-osd: OSD orchestration status for PVC scratch-data-5d42s2 is "completed" 2022-12-03T01:31:55.773 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:55.775359 I | op-osd: creating OSD 4 on PVC "scratch-data-5d42s2" 2022-12-03T01:31:55.773 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:55.775367 I | op-osd: OSD will have its main bluestore block on "scratch-data-5d42s2" 2022-12-03T01:31:56.731 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:56.733844 I | op-osd: OSD orchestration status for PVC scratch-data-3hkxqr is "completed" 2022-12-03T01:31:56.732 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:56.733872 I | op-osd: creating OSD 5 on PVC "scratch-data-3hkxqr" 2022-12-03T01:31:56.733 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:56.733891 I | op-osd: OSD will have its main bluestore block on "scratch-data-3hkxqr" 2022-12-03T01:31:57.240 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.243150 I | op-osd: OSD orchestration status for PVC scratch-data-7zz4lb is "completed" 2022-12-03T01:31:57.241 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.243170 I | op-osd: creating OSD 6 on PVC "scratch-data-7zz4lb" 2022-12-03T01:31:57.241 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.243184 I | op-osd: OSD will have its main bluestore block on "scratch-data-7zz4lb" 2022-12-03T01:31:57.651 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:31:57.942 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.944289 I | op-osd: OSD orchestration status for PVC scratch-data-6vbd4r is "completed" 2022-12-03T01:31:57.942 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.944315 I | op-osd: creating OSD 7 on PVC "scratch-data-6vbd4r" 2022-12-03T01:31:57.942 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:31:57.944329 I | op-osd: OSD will have its main bluestore block on "scratch-data-6vbd4r" 2022-12-03T01:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:31:58.192 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:32:08.193 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:32:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:32:08.789 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:32:08.789 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:32:18.791 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:32:19.489 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:32:19.508 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:32:19.508 INFO:tasks.rook: have 0/12 OSDs 2022-12-03T01:32:29.511 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:32:30.036 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:32:30.058 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:32:30.059 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:32:36.400 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:32:36.402267 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:32:40.060 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:32:40.563 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:32:40.588 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:32:40.588 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:32:50.590 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:32:51.106 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:32:51.129 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:32:51.130 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:01.131 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:01.637 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:01.659 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:01.660 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:11.661 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:12.164 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:12.186 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:12.187 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:22.188 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:22.694 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:22.714 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:22.714 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:32.716 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:33.229 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:33.250 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:33.251 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:36.399 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:33:36.402495 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:33:43.252 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:43.760 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:43.778 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:43.779 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:33:53.780 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:33:54.315 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:33:54.338 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:33:54.339 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:04.341 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:04.853 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:04.872 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:04.873 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:14.874 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:15.384 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:15.403 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:15.404 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:25.405 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:25.912 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:25.931 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:25.932 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:35.933 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:36.401 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:34:36.401871 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:34:36.454 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:36.480 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:36.481 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:46.483 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:46.979 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:47.001 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:47.001 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:34:57.003 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:34:57.512 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:34:57.534 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:34:57.535 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:35:07.538 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:35:08.058 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:35:08.078 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:35:08.078 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:35:18.080 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:35:18.602 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:35:18.633 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:35:18.634 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:35:28.635 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:35:29.180 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:35:29.198 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:35:29.199 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:35:36.404 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:35:36.402148 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:35:39.200 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:35:39.710 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:35:39.735 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:35:39.736 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:35:49.737 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:35:50.235 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:35:50.255 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:35:50.256 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:00.258 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:00.769 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:00.790 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:00.791 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:10.792 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:11.313 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:11.336 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:11.336 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:21.337 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:21.858 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:21.878 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:21.878 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:31.880 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:32.399 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:32.422 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:32.423 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:36.403 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:36:36.401940 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:36:42.425 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:42.936 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:42.980 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:42.981 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:36:52.982 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:36:53.510 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:36:53.531 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:36:53.531 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:03.532 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:04.105 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:04.128 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:04.129 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:14.131 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:14.683 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:14.711 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:14.711 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:24.713 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:25.246 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:25.247 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:35.248 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:35.760 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:35.780 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:35.781 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:36.403 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:37:36.401602 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:37:45.782 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:46.320 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:46.346 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:46.347 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:37:56.348 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:37:56.893 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:37:56.912 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:37:56.912 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:06.913 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:38:07.415 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:38:07.438 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:38:07.439 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:17.441 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:38:17.966 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:38:17.989 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:38:17.990 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:27.992 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:38:28.527 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:38:28.548 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:38:28.548 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:36.403 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:38:36.402063 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:38:38.550 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:38:39.066 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:38:39.085 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:38:39.085 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:49.087 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:38:49.659 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:38:49.685 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:38:49.686 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:38:59.688 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:00.214 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:00.214 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:10.216 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:10.732 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:10.756 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:10.757 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:20.758 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:21.270 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:21.290 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:21.290 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:31.291 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:31.809 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:31.834 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:31.835 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:36.404 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:39:36.402451 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:39:41.836 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:42.325 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:42.351 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:42.351 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:39:52.354 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:39:52.873 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:39:52.886 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:39:52.886 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:02.888 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:03.377 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:03.396 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:03.397 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:13.398 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:13.902 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:13.923 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:13.924 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:23.926 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:24.434 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:24.454 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:24.455 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:34.456 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:34.979 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:34.994 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:34.994 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:36.406 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:40:36.402222 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:40:44.996 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:45.514 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:45.536 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:45.536 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:40:55.538 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:40:56.052 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:40:56.074 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:40:56.075 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:06.077 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:06.585 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:06.604 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:06.604 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:16.606 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:17.154 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:17.178 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:17.178 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:27.180 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:27.687 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:27.707 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:27.707 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:36.405 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:41:36.402030 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:41:37.709 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:38.209 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:38.233 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:38.233 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:48.236 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:48.720 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:48.742 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:48.771 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:41:58.776 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:41:59.299 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:41:59.322 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:41:59.323 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:09.324 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:42:09.840 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:42:09.859 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:42:09.860 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:19.862 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:42:20.370 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:42:20.391 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:42:20.392 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:30.393 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:42:30.899 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:42:30.926 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:42:30.927 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:36.404 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:42:36.402233 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:42:40.929 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:42:41.428 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:42:41.455 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:42:41.455 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:42:51.458 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:42:51.958 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:42:51.981 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:42:51.981 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:01.983 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:02.508 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:02.529 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:02.530 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:12.532 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:13.106 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:13.130 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:13.131 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:23.132 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:23.734 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:23.759 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:23.759 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:33.761 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:34.278 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:34.302 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:34.303 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:36.404 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:43:36.402401 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:43:44.305 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:44.828 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:44.853 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:44.854 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:43:54.855 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:43:55.328 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:43:55.347 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:43:55.347 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:05.349 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:05.875 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:05.899 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:05.900 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:15.901 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:16.410 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:16.434 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:16.435 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:26.436 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:26.928 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:26.950 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:26.951 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:36.405 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:44:36.402558 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:44:36.953 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:37.453 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:37.473 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:37.473 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:47.474 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:48.004 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:48.028 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:48.029 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:44:58.030 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:44:58.534 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:44:58.555 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:44:58.555 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:08.557 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:45:09.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:45:09.103 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:45:09.103 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:19.105 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:45:19.651 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:45:19.676 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:45:19.677 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:29.679 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:45:30.174 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:45:30.198 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:45:30.199 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:36.404 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:45:36.401525 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-12-03T01:45:40.201 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:45:40.707 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:45:40.729 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:45:40.729 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:45:50.730 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:45:51.316 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:45:51.339 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:45:51.340 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:01.342 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:46:01.888 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:46:01.912 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:46:01.913 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:11.915 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:46:12.430 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:46:12.454 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:46:12.455 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:22.456 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:46:22.966 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:46:22.990 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:46:22.991 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:32.992 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph exec rook-ceph-tools-5679b7d8f-68sbf -- ceph osd stat -f json 2022-12-03T01:46:33.481 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:46:33.500 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"num_osds":8,"num_up_osds":8,"osd_up_since":1670031148,"num_in_osds":8,"osd_in_since":1670031112,"num_remapped_pgs":0} 2022-12-03T01:46:33.501 INFO:tasks.rook: have 8/12 OSDs 2022-12-03T01:46:33.502 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:33.504 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 471, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:33.518 DEBUG:teuthology.orchestra.remote:smithi092:rook/deploy/examples/operator.yaml is 26KB 2022-12-03T01:46:33.574 DEBUG:teuthology.orchestra.run.smithi092:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-12-03T01:46:33.656 INFO:teuthology.orchestra.run.smithi092.stdout:deployment.apps "rook-ceph-tools" deleted 2022-12-03T01:46:33.664 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 403, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:33.665 DEBUG:teuthology.orchestra.run.smithi092:> kubectl delete -f cluster.yaml 2022-12-03T01:46:33.748 INFO:teuthology.orchestra.run.smithi092.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-12-03T01:46:33.751 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.748167 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-12-03T01:46:33.751 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.748281 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-12-03T01:46:33.752 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.748364 I | op-osd: context cancelled, exiting OSD update and create loop 2022-12-03T01:46:33.752 INFO:tasks.rook.operator.smithi092.stdout:I1203 01:46:33.748457 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"="context canceled" 2022-12-03T01:46:33.773 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.771124 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-12-03T01:46:33.774 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.771136 I | operator: watching all namespaces for Ceph CRs 2022-12-03T01:46:33.774 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.771150 I | operator: setting up schemes 2022-12-03T01:46:33.775 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.772718 I | ceph-cluster-controller: context cancelled, exiting reconcile 2022-12-03T01:46:33.776 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.773095 I | operator: successfully started the controller-runtime manager 2022-12-03T01:46:33.776 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.773207 I | operator: setting up the controller-runtime manager 2022-12-03T01:46:33.783 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.781068 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-12-03T01:46:33.824 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815299 I | operator: failed to get cert manager 2022-12-03T01:46:33.824 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815345 I | ceph-cluster-controller: successfully started 2022-12-03T01:46:33.825 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815429 I | ceph-cluster-controller: enabling hotplug orchestration 2022-12-03T01:46:33.825 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815460 I | ceph-crashcollector-controller: successfully started 2022-12-03T01:46:33.826 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815500 I | ceph-block-pool-controller: successfully started 2022-12-03T01:46:33.826 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815540 I | ceph-object-store-user-controller: successfully started 2022-12-03T01:46:33.826 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815578 I | ceph-object-realm-controller: successfully started 2022-12-03T01:46:33.827 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815606 I | ceph-object-zonegroup-controller: successfully started 2022-12-03T01:46:33.827 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815642 I | ceph-object-zone-controller: successfully started 2022-12-03T01:46:33.827 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815845 I | ceph-object-controller: successfully started 2022-12-03T01:46:33.827 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815900 I | ceph-file-controller: successfully started 2022-12-03T01:46:33.828 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815949 I | ceph-nfs-controller: successfully started 2022-12-03T01:46:33.828 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.815990 I | ceph-rbd-mirror-controller: successfully started 2022-12-03T01:46:33.828 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816032 I | ceph-client-controller: successfully started 2022-12-03T01:46:33.828 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816087 I | ceph-filesystem-mirror-controller: successfully started 2022-12-03T01:46:33.829 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816135 I | operator: rook-ceph-operator-config-controller successfully started 2022-12-03T01:46:33.829 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816162 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-12-03T01:46:33.829 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816204 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-12-03T01:46:33.829 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816233 I | ceph-bucket-topic: successfully started 2022-12-03T01:46:33.829 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816258 I | ceph-bucket-notification: successfully started 2022-12-03T01:46:33.830 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816277 I | ceph-bucket-notification: successfully started 2022-12-03T01:46:33.830 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816297 I | ceph-fs-subvolumegroup-controller: successfully started 2022-12-03T01:46:33.830 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.816318 I | blockpool-rados-namespace-controller: successfully started 2022-12-03T01:46:33.830 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.817564 I | operator: starting the controller-runtime manager 2022-12-03T01:46:33.926 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.922393 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-12-03T01:46:33.926 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.922441 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-12-03T01:46:33.927 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.922456 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-12-03T01:46:33.928 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.923385 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-12-03T01:46:33.968 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.965444 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap) 2022-12-03T01:46:33.968 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:33.965466 I | operator: rook-ceph-operator-config-controller done reconciling 2022-12-03T01:46:34.007 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.005090 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" can be deleted safely. deleting CephCluster "rook-ceph/rook-ceph" 2022-12-03T01:46:34.025 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.022707 I | ceph-cluster-controller: cleaning up CephCluster "rook-ceph/rook-ceph" 2022-12-03T01:46:34.025 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.022747 I | ceph-cluster-controller: checking any PVC created by drivers "rook-ceph.cephfs.csi.ceph.com" and "rook-ceph.rbd.csi.ceph.com" with clusterID "rook-ceph" 2022-12-03T01:46:34.049 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044442 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi092-lv-1" 2022-12-03T01:46:34.049 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044464 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi092-lv-2" 2022-12-03T01:46:34.050 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044471 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi092-lv-3" 2022-12-03T01:46:34.050 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044478 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi092-lv-4" 2022-12-03T01:46:34.050 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044484 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi121-lv-1" 2022-12-03T01:46:34.050 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044489 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi121-lv-2" 2022-12-03T01:46:34.051 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044495 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi121-lv-3" 2022-12-03T01:46:34.051 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044501 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi121-lv-4" 2022-12-03T01:46:34.051 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044509 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi138-lv-1" 2022-12-03T01:46:34.051 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044515 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi138-lv-2" 2022-12-03T01:46:34.052 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044521 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi138-lv-3" 2022-12-03T01:46:34.052 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044528 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi138-lv-4" 2022-12-03T01:46:34.052 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044535 I | ceph-cluster-controller: no volume attachments for cluster "rook-ceph" 2022-12-03T01:46:34.052 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.044572 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon" 2022-12-03T01:46:34.055 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.053641 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon-endpoints" 2022-12-03T01:46:34.064 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.062256 I | ceph-spec: removing finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-12-03T01:46:34.109 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.106998 E | clusterdisruption-controller: cephcluster "rook-ceph/rook-ceph" seems to be deleted, not requeuing until triggered again 2022-12-03T01:46:34.111 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.107831 I | ceph-csi: CSI Ceph RBD driver disabled 2022-12-03T01:46:34.111 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.107851 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-12-03T01:46:34.114 INFO:tasks.rook:Waiting for cluster to stop 2022-12-03T01:46:34.116 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:46:34.135 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.133086 I | op-k8sutil: Removed daemonset csi-rbdplugin 2022-12-03T01:46:34.139 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.137369 I | ceph-spec: object "rook-ceph-mgr-a-keyring" matched on delete, reconciling 2022-12-03T01:46:34.149 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.146660 I | op-k8sutil: "csi-rbdplugin" still found. waiting... 2022-12-03T01:46:34.171 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.166750 I | ceph-spec: object "rook-ceph-admin-keyring" matched on delete, reconciling 2022-12-03T01:46:34.174 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.169124 I | ceph-spec: object "rook-ceph-osd-5" matched on delete, reconciling 2022-12-03T01:46:34.175 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.172069 I | ceph-spec: object "rook-ceph-mon-a" matched on delete, reconciling 2022-12-03T01:46:34.277 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.274499 I | ceph-spec: object "rook-ceph-osd-4" matched on delete, reconciling 2022-12-03T01:46:34.290 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.287662 I | ceph-spec: object "rook-ceph-crashcollector-smithi121" matched on delete, reconciling 2022-12-03T01:46:34.311 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.308018 I | ceph-spec: object "rook-csi-rbd-provisioner" matched on delete, reconciling 2022-12-03T01:46:34.313 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.311030 I | ceph-spec: object "rook-ceph-osd-6" matched on delete, reconciling 2022-12-03T01:46:34.334 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:46:34.334 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 17m 2022-12-03T01:46:34.335 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 17m 2022-12-03T01:46:34.335 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 17m 2022-12-03T01:46:34.335 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 17m 2022-12-03T01:46:34.335 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-68fxq 2/2 Running 0 17m 2022-12-03T01:46:34.336 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-jkkc9 2/2 Running 0 17m 2022-12-03T01:46:34.336 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 17m 2022-12-03T01:46:34.336 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 17m 2022-12-03T01:46:34.336 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-crashcollector-smithi121-5987777896-mr244 1/1 Running 0 14m 2022-12-03T01:46:34.337 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-crashcollector-smithi138-5f57b7966b-zccgh 1/1 Running 0 15m 2022-12-03T01:46:34.337 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mgr-a-699657dbc-x8xp5 1/1 Running 0 15m 2022-12-03T01:46:34.337 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Running 0 16m 2022-12-03T01:46:34.337 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-6668c88b8d-2h5f2 1/1 Running 0 15m 2022-12-03T01:46:34.338 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-66bd7d5784-f5xkn 1/1 Running 0 15m 2022-12-03T01:46:34.338 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 19m 2022-12-03T01:46:34.338 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-0-5445dfd566-8f7sh 1/1 Running 0 14m 2022-12-03T01:46:34.338 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-1-6dc74cb9b-mp99f 1/1 Running 0 14m 2022-12-03T01:46:34.339 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-2-86ddbf6c4c-5rnjh 1/1 Running 0 14m 2022-12-03T01:46:34.339 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-3-75487d4b8b-v426c 1/1 Running 0 14m 2022-12-03T01:46:34.339 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-4-86db467b6d-qvd2s 1/1 Running 0 14m 2022-12-03T01:46:34.339 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-5-5bd654db8d-5vdzg 1/1 Running 0 14m 2022-12-03T01:46:34.340 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-6-797f679775-p6nrm 1/1 Running 0 14m 2022-12-03T01:46:34.340 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-7-6f6df46bdd-8m8cb 1/1 Running 0 14m 2022-12-03T01:46:34.340 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-0nxlcm-5v9s2 0/1 Completed 0 15m 2022-12-03T01:46:34.340 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-10vzmld-rs4mn 0/1 Pending 0 14m 2022-12-03T01:46:34.341 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-11m4fvt-bgpsq 0/1 Pending 0 14m 2022-12-03T01:46:34.341 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-1gl9wr-7nmrq 0/1 Completed 0 15m 2022-12-03T01:46:34.341 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-2hkmfs-bmbbm 0/1 Completed 0 15m 2022-12-03T01:46:34.341 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-3hkxqr-drlgj 0/1 Completed 0 15m 2022-12-03T01:46:34.342 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-45wsd7-gg62c 0/1 Completed 0 15m 2022-12-03T01:46:34.342 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-5d42s2-jh6x5 0/1 Completed 0 15m 2022-12-03T01:46:34.343 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-6vbd4r-rb97k 0/1 Completed 0 15m 2022-12-03T01:46:34.343 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-7zz4lb-6lbft 0/1 Completed 0 15m 2022-12-03T01:46:34.344 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-8zz8z7-hdhrs 0/1 Pending 0 14m 2022-12-03T01:46:34.344 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-prepare-scratch-data-9r4fv9-wtbb9 0/1 Pending 0 14m 2022-12-03T01:46:34.344 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Terminating 0 18m 2022-12-03T01:46:34.991 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:34.988797 I | ceph-spec: object "rook-ceph-mgr-a" matched on delete, reconciling 2022-12-03T01:46:35.025 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.023041 I | ceph-spec: object "rook-ceph-mon-endpoints" matched on delete, reconciling 2022-12-03T01:46:35.090 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.087914 I | ceph-spec: object "rook-ceph-mon" matched on delete, reconciling 2022-12-03T01:46:35.184 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.181712 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-12-03T01:46:35.270 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.266994 I | ceph-spec: object "rook-ceph-mons-keyring" matched on delete, reconciling 2022-12-03T01:46:35.271 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.269026 I | ceph-spec: object "rook-ceph-mon-c" matched on delete, reconciling 2022-12-03T01:46:35.292 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.290088 I | ceph-spec: object "rook-ceph-osd-1" matched on delete, reconciling 2022-12-03T01:46:35.329 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.325107 I | ceph-spec: object "rook-ceph-config" matched on delete, reconciling 2022-12-03T01:46:35.376 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.373365 I | ceph-spec: object "rook-csi-cephfs-node" matched on delete, reconciling 2022-12-03T01:46:35.452 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.449923 I | ceph-spec: object "cluster-peer-token-rook-ceph" matched on delete, reconciling 2022-12-03T01:46:35.991 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:35.989040 I | ceph-spec: object "rook-ceph-mgr" matched on delete, reconciling 2022-12-03T01:46:36.024 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.022279 I | ceph-spec: object "rook-csi-cephfs-provisioner" matched on delete, reconciling 2022-12-03T01:46:36.075 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.073402 I | ceph-spec: object "rook-ceph-osd-7" matched on delete, reconciling 2022-12-03T01:46:36.108 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.106152 I | ceph-spec: object "rook-ceph-mon-b" matched on delete, reconciling 2022-12-03T01:46:36.201 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.198828 I | ceph-spec: object "rook-ceph-osd-2" matched on delete, reconciling 2022-12-03T01:46:36.234 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.232018 I | ceph-spec: object "rook-ceph-crashcollector-smithi138" matched on delete, reconciling 2022-12-03T01:46:36.241 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.238916 I | ceph-spec: object "rook-ceph-crash-collector-keyring" matched on delete, reconciling 2022-12-03T01:46:36.300 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.298392 I | ceph-spec: object "rook-ceph-osd-3" matched on delete, reconciling 2022-12-03T01:46:36.316 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:36.313912 I | ceph-spec: object "rook-csi-rbd-node" matched on delete, reconciling 2022-12-03T01:46:39.346 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:46:39.489 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:46:39.490 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Running 0 17m 2022-12-03T01:46:39.490 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 17m 2022-12-03T01:46:39.491 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 17m 2022-12-03T01:46:39.491 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Running 0 17m 2022-12-03T01:46:39.491 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zcdj 5/5 Running 0 17m 2022-12-03T01:46:39.492 INFO:teuthology.orchestra.run.smithi092.stdout:csi-rbdplugin-provisioner-56d69f5d8-4zh27 5/5 Running 0 17m 2022-12-03T01:46:39.492 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-crashcollector-smithi121-5987777896-mr244 1/1 Terminating 0 14m 2022-12-03T01:46:39.493 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-crashcollector-smithi138-5f57b7966b-zccgh 1/1 Terminating 0 15m 2022-12-03T01:46:39.493 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mgr-a-699657dbc-x8xp5 1/1 Terminating 0 15m 2022-12-03T01:46:39.494 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-a-65f8f96ddd-l4nqr 1/1 Terminating 0 16m 2022-12-03T01:46:39.494 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-b-6668c88b8d-2h5f2 1/1 Terminating 0 16m 2022-12-03T01:46:39.494 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-mon-c-66bd7d5784-f5xkn 1/1 Terminating 0 15m 2022-12-03T01:46:39.495 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 19m 2022-12-03T01:46:39.495 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-0-5445dfd566-8f7sh 1/1 Terminating 0 14m 2022-12-03T01:46:39.495 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-1-6dc74cb9b-mp99f 1/1 Terminating 0 14m 2022-12-03T01:46:39.496 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-2-86ddbf6c4c-5rnjh 1/1 Terminating 0 14m 2022-12-03T01:46:39.496 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-3-75487d4b8b-v426c 1/1 Terminating 0 14m 2022-12-03T01:46:39.496 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-4-86db467b6d-qvd2s 1/1 Terminating 0 14m 2022-12-03T01:46:39.496 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-5-5bd654db8d-5vdzg 1/1 Terminating 0 14m 2022-12-03T01:46:39.497 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-6-797f679775-p6nrm 1/1 Terminating 0 14m 2022-12-03T01:46:39.497 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-7-6f6df46bdd-8m8cb 1/1 Terminating 0 14m 2022-12-03T01:46:39.497 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Terminating 0 18m 2022-12-03T01:46:40.172 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:40.169300 I | op-k8sutil: confirmed csi-rbdplugin does not exist 2022-12-03T01:46:40.172 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:40.169329 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-12-03T01:46:40.199 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:40.195602 I | op-k8sutil: Removed deployment csi-rbdplugin-provisioner 2022-12-03T01:46:40.203 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:40.201143 I | op-k8sutil: "csi-rbdplugin-provisioner" still found. waiting... 2022-12-03T01:46:44.211 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.209281 I | op-k8sutil: confirmed csi-rbdplugin-provisioner does not exist 2022-12-03T01:46:44.227 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.224630 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-12-03T01:46:44.228 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.224650 I | ceph-csi: CSI CephFS driver disabled 2022-12-03T01:46:44.228 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.224656 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-12-03T01:46:44.340 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.337074 I | op-k8sutil: Removed daemonset csi-cephfsplugin 2022-12-03T01:46:44.343 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:44.341003 I | op-k8sutil: "csi-cephfsplugin" still found. waiting... 2022-12-03T01:46:44.500 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:46:44.602 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:46:44.603 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-649zw 2/2 Terminating 0 17m 2022-12-03T01:46:44.603 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Running 0 17m 2022-12-03T01:46:44.604 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Running 0 17m 2022-12-03T01:46:44.604 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-xnhr5 2/2 Terminating 0 17m 2022-12-03T01:46:44.604 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 19m 2022-12-03T01:46:44.604 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-0-5445dfd566-8f7sh 1/1 Terminating 0 14m 2022-12-03T01:46:44.604 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-1-6dc74cb9b-mp99f 1/1 Terminating 0 14m 2022-12-03T01:46:44.605 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-2-86ddbf6c4c-5rnjh 1/1 Terminating 0 14m 2022-12-03T01:46:44.605 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-3-75487d4b8b-v426c 1/1 Terminating 0 14m 2022-12-03T01:46:44.605 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-4-86db467b6d-qvd2s 1/1 Terminating 0 14m 2022-12-03T01:46:44.605 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-5-5bd654db8d-5vdzg 1/1 Terminating 0 14m 2022-12-03T01:46:44.606 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-6-797f679775-p6nrm 1/1 Terminating 0 14m 2022-12-03T01:46:44.606 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-osd-7-6f6df46bdd-8m8cb 1/1 Terminating 0 14m 2022-12-03T01:46:44.606 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Terminating 0 18m 2022-12-03T01:46:48.350 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:48.348359 I | op-k8sutil: confirmed csi-cephfsplugin does not exist 2022-12-03T01:46:48.351 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:48.348385 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-12-03T01:46:48.375 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:48.372981 I | op-k8sutil: Removed deployment csi-cephfsplugin-provisioner 2022-12-03T01:46:48.379 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:48.377858 I | op-k8sutil: "csi-cephfsplugin-provisioner" still found. waiting... 2022-12-03T01:46:49.608 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph get pods 2022-12-03T01:46:49.743 INFO:teuthology.orchestra.run.smithi092.stdout:NAME READY STATUS RESTARTS AGE 2022-12-03T01:46:49.743 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-4ds6n 5/5 Terminating 0 17m 2022-12-03T01:46:49.743 INFO:teuthology.orchestra.run.smithi092.stdout:csi-cephfsplugin-provisioner-75875b5887-9rb5q 5/5 Terminating 0 17m 2022-12-03T01:46:49.744 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-operator-5ddcc89457-mhcgp 1/1 Running 0 19m 2022-12-03T01:46:49.744 INFO:teuthology.orchestra.run.smithi092.stdout:rook-ceph-tools-5679b7d8f-68sbf 1/1 Terminating 0 18m 2022-12-03T01:46:49.745 DEBUG:teuthology.orchestra.run.smithi092:> kubectl -n rook-ceph delete configmap rook-config-override 2022-12-03T01:46:49.836 INFO:teuthology.orchestra.run.smithi092.stdout:configmap "rook-config-override" deleted 2022-12-03T01:46:49.865 DEBUG:teuthology.orchestra.run.smithi092:> rm -f cluster.yaml 2022-12-03T01:46:49.873 INFO:tasks.rook:Checking cluster log for badness... 2022-12-03T01:46:49.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/lib/rook/rook-ceph/log/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2022-12-03T01:46:49.929 INFO:teuthology.orchestra.run.smithi092.stderr:grep: /var/lib/rook/rook-ceph/log/ceph.log: No such file or directory 2022-12-03T01:46:49.931 INFO:tasks.rook:Compressing logs... 2022-12-03T01:46:49.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:46:49.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:46:49.982 DEBUG:teuthology.orchestra.run.smithi138:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:46:49.991 INFO:teuthology.orchestra.run.smithi092.stderr:find: '/var/lib/rook/rook-ceph/log': No such file or directory 2022-12-03T01:46:50.955 INFO:tasks.rook:Archiving logs... 2022-12-03T01:46:50.957 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/ubuntu@smithi092.front.sepia.ceph.com/log 2022-12-03T01:46:50.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:46:50.970 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/rook/rook-ceph/log: Cannot open: No such file or directory 2022-12-03T01:46:50.971 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2022-12-03T01:46:50.974 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/ubuntu@smithi121.front.sepia.ceph.com/log 2022-12-03T01:46:50.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:46:51.241 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/ubuntu@smithi138.front.sepia.ceph.com/log 2022-12-03T01:46:51.243 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-12-03T01:46:51.445 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:51.445 INFO:tasks.rook:Cleaning up rook operator 2022-12-03T01:46:51.446 DEBUG:teuthology.orchestra.run.smithi092:> kubectl delete -f operator.yaml 2022-12-03T01:46:51.519 INFO:teuthology.orchestra.run.smithi092.stdout:configmap "rook-ceph-operator-config" deleted 2022-12-03T01:46:51.527 INFO:teuthology.orchestra.run.smithi092.stdout:deployment.apps "rook-ceph-operator" deleted 2022-12-03T01:46:51.536 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf rook operator.yaml 2022-12-03T01:46:51.668 INFO:tasks.rook.operator.smithi092.stdout:2022-12-03 01:46:51.665838 I | operator: shutdown signal received, exiting... context canceled 2022-12-03T01:46:52.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf /var/lib/rook 2022-12-03T01:46:52.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf /var/lib/rook 2022-12-03T01:46:52.720 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf /var/lib/rook 2022-12-03T01:46:52.743 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:52.892 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c8eea24934af4876a500584d87de2c73 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_48d203d9b836cc224eb4085f6198d948c6fa1bb1/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-12-03T01:46:52.895 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-12-03T01:46:52.908 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2022-12-03T01:46:52.959 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi092:/dev/vg_nvme/lv_1... 2022-12-03T01:46:52.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo nvme disconnect -n lv_1 2022-12-03T01:46:53.267 INFO:teuthology.orchestra.run.smithi092.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:46:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:53.270 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi092:/dev/vg_nvme/lv_2... 2022-12-03T01:46:53.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo nvme disconnect -n lv_2 2022-12-03T01:46:53.583 INFO:teuthology.orchestra.run.smithi092.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:46:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:53.586 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi092:/dev/vg_nvme/lv_3... 2022-12-03T01:46:53.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo nvme disconnect -n lv_3 2022-12-03T01:46:53.877 INFO:teuthology.orchestra.run.smithi092.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:46:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:53.886 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi092:/dev/vg_nvme/lv_4... 2022-12-03T01:46:53.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo nvme disconnect -n lv_4 2022-12-03T01:46:54.167 INFO:teuthology.orchestra.run.smithi092.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:46:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:54.170 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2022-12-03T01:46:54.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/scratch_devs 2022-12-03T01:46:54.185 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi121:/dev/vg_nvme/lv_1... 2022-12-03T01:46:54.186 DEBUG:teuthology.orchestra.run.smithi121:> sudo nvme disconnect -n lv_1 2022-12-03T01:46:54.461 INFO:teuthology.orchestra.run.smithi121.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:46:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:54.463 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi121:/dev/vg_nvme/lv_2... 2022-12-03T01:46:54.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo nvme disconnect -n lv_2 2022-12-03T01:46:54.761 INFO:teuthology.orchestra.run.smithi121.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:46:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:54.764 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi121:/dev/vg_nvme/lv_3... 2022-12-03T01:46:54.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo nvme disconnect -n lv_3 2022-12-03T01:46:55.093 INFO:teuthology.orchestra.run.smithi121.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:46:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:55.096 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi121:/dev/vg_nvme/lv_4... 2022-12-03T01:46:55.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo nvme disconnect -n lv_4 2022-12-03T01:46:55.397 INFO:teuthology.orchestra.run.smithi121.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:46:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:55.399 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-12-03T01:46:55.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/scratch_devs 2022-12-03T01:46:55.415 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_1... 2022-12-03T01:46:55.416 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_1 2022-12-03T01:46:55.737 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-12-03T01:46:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:55.740 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_2... 2022-12-03T01:46:55.740 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_2 2022-12-03T01:46:56.045 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-12-03T01:46:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:56.047 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_3... 2022-12-03T01:46:56.048 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_3 2022-12-03T01:46:56.337 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-12-03T01:46:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:56.339 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi138:/dev/vg_nvme/lv_4... 2022-12-03T01:46:56.340 DEBUG:teuthology.orchestra.run.smithi138:> sudo nvme disconnect -n lv_4 2022-12-03T01:46:56.629 INFO:teuthology.orchestra.run.smithi138.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-12-03T01:46:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:46:56.632 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2022-12-03T01:46:56.633 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/scratch_devs 2022-12-03T01:46:56.650 DEBUG:teuthology.run_tasks:Unwinding manager kubeadm 2022-12-03T01:46:56.661 INFO:tasks.kubeadm:Tearing down kubeadm 2022-12-03T01:46:56.662 DEBUG:teuthology.orchestra.run.smithi092:> kubectl delete -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-12-03T01:46:56.877 INFO:teuthology.orchestra.run.smithi092.stdout:namespace "kube-flannel" deleted 2022-12-03T01:46:56.893 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrole.rbac.authorization.k8s.io "flannel" deleted 2022-12-03T01:46:56.910 INFO:teuthology.orchestra.run.smithi092.stdout:clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted 2022-12-03T01:46:56.945 INFO:teuthology.orchestra.run.smithi092.stdout:serviceaccount "flannel" deleted 2022-12-03T01:46:57.013 INFO:teuthology.orchestra.run.smithi092.stdout:configmap "kube-flannel-cfg" deleted 2022-12-03T01:46:57.035 INFO:teuthology.orchestra.run.smithi092.stdout:daemonset.apps "kube-flannel-ds" deleted 2022-12-03T01:47:02.712 INFO:tasks.kubeadm:Deconfiguring kubectl 2022-12-03T01:47:02.713 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:47:02.732 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:47:02.746 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf .kube && sudo rm -rf /root/.kube 2022-12-03T01:47:02.761 INFO:tasks.kubeadm:Cleaning up node 2022-12-03T01:47:02.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:47:02.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:47:02.790 DEBUG:teuthology.orchestra.run.smithi138:> sudo kubeadm reset cleanup-node -f 2022-12-03T01:47:02.821 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Reading configuration from the cluster... 2022-12-03T01:47:02.822 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-12-03T01:47:02.825 INFO:teuthology.orchestra.run.smithi121.stdout:[preflight] Running pre-flight checks 2022-12-03T01:47:02.826 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] No etcd config found. Assuming external etcd 2022-12-03T01:47:02.826 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-12-03T01:47:02.827 INFO:teuthology.orchestra.run.smithi121.stderr:W1203 01:47:02.823142 53819 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-12-03T01:47:02.828 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Stopping the kubelet service 2022-12-03T01:47:02.853 INFO:teuthology.orchestra.run.smithi092.stdout:[preflight] Running pre-flight checks 2022-12-03T01:47:02.854 INFO:teuthology.orchestra.run.smithi138.stdout:[preflight] Running pre-flight checks 2022-12-03T01:47:02.855 INFO:teuthology.orchestra.run.smithi138.stderr:W1203 01:47:02.851067 49237 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-12-03T01:47:02.857 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] No etcd config found. Assuming external etcd 2022-12-03T01:47:02.858 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-12-03T01:47:02.858 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Stopping the kubelet service 2022-12-03T01:47:02.880 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Stopping the kubelet service 2022-12-03T01:47:02.899 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:47:02.920 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:47:02.939 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-12-03T01:47:05.036 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:47:05.037 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:47:05.037 INFO:teuthology.orchestra.run.smithi138.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet] 2022-12-03T01:47:05.038 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:47:05.039 INFO:teuthology.orchestra.run.smithi138.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:47:05.039 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:47:05.039 INFO:teuthology.orchestra.run.smithi138.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:47:05.040 INFO:teuthology.orchestra.run.smithi138.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:47:05.040 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:47:05.040 INFO:teuthology.orchestra.run.smithi138.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:47:05.041 INFO:teuthology.orchestra.run.smithi138.stdout:to reset your system's IPVS tables. 2022-12-03T01:47:05.041 INFO:teuthology.orchestra.run.smithi138.stdout: 2022-12-03T01:47:05.041 INFO:teuthology.orchestra.run.smithi138.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:47:05.042 INFO:teuthology.orchestra.run.smithi138.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:47:07.788 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:47:07.788 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:47:07.789 INFO:teuthology.orchestra.run.smithi121.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet] 2022-12-03T01:47:07.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:47:07.792 INFO:teuthology.orchestra.run.smithi121.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:47:07.792 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:47:07.793 INFO:teuthology.orchestra.run.smithi121.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:47:07.794 INFO:teuthology.orchestra.run.smithi121.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:47:07.794 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:47:07.795 INFO:teuthology.orchestra.run.smithi121.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:47:07.795 INFO:teuthology.orchestra.run.smithi121.stdout:to reset your system's IPVS tables. 2022-12-03T01:47:07.796 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-12-03T01:47:07.797 INFO:teuthology.orchestra.run.smithi121.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:47:07.798 INFO:teuthology.orchestra.run.smithi121.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:47:08.707 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-12-03T01:47:08.708 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-12-03T01:47:08.709 INFO:teuthology.orchestra.run.smithi092.stdout:[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet] 2022-12-03T01:47:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:47:08.719 INFO:teuthology.orchestra.run.smithi092.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-12-03T01:47:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:47:08.720 INFO:teuthology.orchestra.run.smithi092.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-12-03T01:47:08.721 INFO:teuthology.orchestra.run.smithi092.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-12-03T01:47:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:47:08.722 INFO:teuthology.orchestra.run.smithi092.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-12-03T01:47:08.722 INFO:teuthology.orchestra.run.smithi092.stdout:to reset your system's IPVS tables. 2022-12-03T01:47:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: 2022-12-03T01:47:08.723 INFO:teuthology.orchestra.run.smithi092.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-12-03T01:47:08.724 INFO:teuthology.orchestra.run.smithi092.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-12-03T01:47:08.725 INFO:tasks.kubeadm:Uninstalling kube{adm,let,ctl} 2022-12-03T01:47:08.726 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-12-03T01:47:08.764 INFO:teuthology.task.clock:Checking final clock skew... 2022-12-03T01:47:08.765 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:47:08.770 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:47:08.776 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-12-03T01:47:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:47:08.790 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2022-12-03T01:47:08.790 INFO:teuthology.orchestra.run.smithi092.stdout:+hv01.front.sepi 216.240.36.24 3 u 3 64 377 0.094 0.566 1.111 2022-12-03T01:47:08.791 INFO:teuthology.orchestra.run.smithi092.stdout:*hv02.front.sepi 163.237.218.18 2 u 54 64 377 0.089 0.080 0.700 2022-12-03T01:47:08.791 INFO:teuthology.orchestra.run.smithi092.stdout:+hv03.front.sepi 206.82.28.3 3 u 66 64 377 0.086 -0.373 0.941 2022-12-03T01:47:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:47:08.794 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:47:08.794 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-12-03T01:47:08.795 INFO:teuthology.orchestra.run.smithi121.stdout:+hv01.front.sepi 216.240.36.24 3 u 60 64 377 0.089 1.640 0.323 2022-12-03T01:47:08.795 INFO:teuthology.orchestra.run.smithi121.stdout:*hv02.front.sepi 163.237.218.18 2 u 53 64 377 0.080 0.883 0.397 2022-12-03T01:47:08.795 INFO:teuthology.orchestra.run.smithi121.stdout:+hv03.front.sepi 206.82.28.3 3 u 62 64 377 0.086 0.459 0.391 2022-12-03T01:47:08.796 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:47:08.801 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2022-12-03T01:47:08.802 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2022-12-03T01:47:08.802 INFO:teuthology.orchestra.run.smithi138.stdout:+hv01.front.sepi 216.240.36.24 3 u 59 64 377 0.079 0.858 0.053 2022-12-03T01:47:08.803 INFO:teuthology.orchestra.run.smithi138.stdout:*hv02.front.sepi 163.237.218.18 2 u 51 64 377 0.087 -0.081 0.034 2022-12-03T01:47:08.803 INFO:teuthology.orchestra.run.smithi138.stdout:+hv03.front.sepi 206.82.28.3 3 u 62 64 377 0.113 -0.055 0.048 2022-12-03T01:47:08.804 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-12-03T01:47:08.804 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-12-03T01:47:08.821 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-12-03T01:47:08.822 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-12-03T01:47:08.861 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-12-03T01:47:08.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-12-03T01:47:08.956 INFO:teuthology.task.internal:Duration was 1625.173725 seconds 2022-12-03T01:47:08.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-12-03T01:47:09.006 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-12-03T01:47:09.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:47:09.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:47:09.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-12-03T01:47:09.072 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-12-03T01:47:09.073 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2022-12-03T01:47:09.075 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:47:09.119 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2022-12-03T01:47:09.120 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:47:09.138 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2022-12-03T01:47:09.140 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-12-03T01:47:09.165 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-12-03T01:47:09.166 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:47:09.170 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:47:09.194 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-12-03T01:47:09.259 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-12-03T01:47:09.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:47:09.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:47:09.287 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-12-03T01:47:09.395 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-12-03T01:47:09.418 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-12-03T01:47:09.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:47:09.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:47:09.460 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-12-03T01:47:09.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-12-03T01:47:09.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.496 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.506 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.510 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2022-12-03T01:47:09.518 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2022-12-03T01:47:09.538 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2022-12-03T01:47:09.570 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:47:09.590 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:47:09.598 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2022-12-03T01:47:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-12-03T01:47:09.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-12-03T01:47:09.638 INFO:teuthology.task.internal:Transferring archived files... 2022-12-03T01:47:09.640 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/smithi092 2022-12-03T01:47:09.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:47:09.689 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/smithi121 2022-12-03T01:47:09.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:47:09.748 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682/remote/smithi138 2022-12-03T01:47:09.750 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-12-03T01:47:09.800 INFO:teuthology.task.internal:Removing archive directory... 2022-12-03T01:47:09.800 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:47:09.804 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:47:09.808 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2022-12-03T01:47:09.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-12-03T01:47:09.869 INFO:teuthology.task.internal:Not uploading archives. 2022-12-03T01:47:09.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-12-03T01:47:09.892 INFO:teuthology.task.internal:Tidying up after the test... 2022-12-03T01:47:09.893 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:47:09.896 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:47:09.900 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-12-03T01:47:09.901 INFO:teuthology.orchestra.run.smithi092.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:47 /home/ubuntu/cephtest 2022-12-03T01:47:09.905 INFO:teuthology.orchestra.run.smithi121.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:47 /home/ubuntu/cephtest 2022-12-03T01:47:09.908 INFO:teuthology.orchestra.run.smithi138.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 01:47 /home/ubuntu/cephtest 2022-12-03T01:47:09.911 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2022-12-03T01:47:10.004 INFO:teuthology.nuke:Checking targets against current locks 2022-12-03T01:47:10.042 DEBUG:teuthology.nuke:shortname: smithi092 2022-12-03T01:47:10.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:47:10.084 DEBUG:teuthology.nuke:shortname: smithi121 2022-12-03T01:47:10.085 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:47:10.097 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.573656', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEY+i4wrK2zvejhB63LailoMq8pVSff2L0VUWp4HU07jR0KeoghN2mxvuvxPb3VyJ5FopueYw7HBahS+oiPLNozJU8PtNTmC1WPHLDUZ6UlAkBGhNevPhTWhjE43PsYQZRV4wRYvDIiEtvBJhydOl6GB+6MP5KKiIL3ySYchpkCxdXVhxjHqMaDe/P8UOTlWmaM6MEuQfizBRDbvFQyfow/1hJ7iek2v4NQTk130hvCjIKEJzyRSXWG4KPg54mpTomWTCxKOBBBtLVrXHQo+7l1TzTOdXcCIfZeux15uoNBDFNT2MeCMqTf6jzfannR0pJwPwQ6cD/USvejLEh4s3ySjjX0WCJQMnbKMQy3qAwNjcrxAPu+wP9d/A520rrroJNURgI+z4X8bzifCpjZ2l0zRmJ/HqEzJyaqcRF3Ldh1SEElUNVb2ukB83DJudofvmu8ypG/BBbLKx/RTNrFEBoySlZs4FNwm9SL1AbIrLCRi6A4mBVn6/riSvCGCZySf8='} 2022-12-03T01:47:10.116 DEBUG:teuthology.nuke:shortname: smithi138 2022-12-03T01:47:10.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-03T01:47:10.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.572369', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIvh7ryYHiaQh2Dqw+PCcM93p9ca0Rf/8CP4kp7w9y0oJ6/XIH/LXXXWDl4vT1LKHo/7YJCUq9l0xy0+oAjvuNYMJJEtbou64msidOTPLaEwkbLlbwopnNKCp5MrcHQ9Jsi8O0XXMyqMgZbjaUHeMhARdK/H3CVc1hlxtpgl2D+7BgNHu20/AwAI0EzqluDKlF7ghhCJOAnIt4SACBe777/yYYSDdVrsVHh4NEDBahtM7MVd3gsO0lZOJaE+eCh5cQ+ieaZ1T1igpV4b3AzuJyivBq1UdcIMRyJ9dMHzp7DvahSLwT8awr8LtHFyip7PZlifq7r1KqJIedb6oH8Xr7+TVYCBLblvJxyG7Q0qos9pfzQFQ2Yri6g8mm2d6f2RcgXQatyaoaoKh2IBkikwL81TqotZSocGex0ze7LcnNaeLSoDaGPOb2/mvvB16sMYBu1fO+Eyc4O8NSnuidKg9sg79wEZC7akoLgLmCLgLvZ9w0P42Z9MBQ2s1uWjg4ths='} 2022-12-03T01:47:10.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2022-12-02_15:43:15-rados-wip-yuri7-testing-2022-11-18-1548-distro-default-smithi/7101682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-12-03 01:11:16.570268', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/hO+jEDXI7jJAhwTvzKqeELuuMek2Mj/hs6PWGfAWgX0xYAIRkediizPkh8kfDtWym2QannL8tZjtgmC1thvZ4+eHF6Zx6Gv4vAfh4uONdV74Iql6lwTLCS+q+kQ7ZXBvn4pNQj+/YCgGhb6O61pQ3Bbjy0NDJYhWQ4rpvF3HhKoYdhkmDXzAU6iDLQ8n4Bh6nV1K9icCLsYcnQi+roxTIT+QMIPDWimXR7yaRvd8ZNTv6A+C+zGeQu8gzFNBSkWWqmFoORqnLu2GKW6JlxSWEFT1IIAZUc0zg8Tig+N6qg33KwJmU46QP0x7wbIACqrId9n8YiIlqhPIU3z5YG5KIqaeD6QS9gZNQZPI6zDQSSTLOkO45l73SORGGGBxZ/gojMyF94mfD7Uj7klpeZid6MpKbs0OXCvZgVAKsFbPvy7FV9tODC6LUG79ArL1C65h9DT22zUYKff2+wKDQpYV/5tBJyVMPd/Edx6tT7mkRyMeIL5aV9rkkTk4B6C1bY8='} 2022-12-03T01:47:10.165 INFO:teuthology.orchestra.console:Power off smithi092 2022-12-03T01:47:10.167 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:47:10.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:10.245 INFO:teuthology.orchestra.console:Power off smithi121 2022-12-03T01:47:10.247 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:47:10.279 INFO:teuthology.orchestra.console:Power off smithi138 2022-12-03T01:47:10.281 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-12-03T01:47:10.300 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:10.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:14.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:14.339 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:14.359 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:18.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:18.464 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:18.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:18.696 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2022-12-03T01:47:22.567 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:22.580 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-12-03T01:47:22.782 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2022-12-03T01:47:22.802 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2022-12-03T01:47:22.911 INFO:teuthology.run:Summary data: description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} duration: 1625.1737246513367 failure_reason: '''check osd count'' reached maximum tries (90) after waiting for 900 seconds' owner: scheduled_lflores@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c8eea24934af4876a500584d87de2c73 status: fail success: false 2022-12-03T01:47:22.912 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-12-03T01:47:22.981 INFO:teuthology.run:FAIL