2022-07-21T20:16:36.304 INFO:root:teuthology version: 1.1.1.dev389+g8d598431 2022-07-21T20:16:36.307 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} --verbose --owner scheduled_dgalloway@teuthology --name dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi --archive /home/teuthworker/archive/dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi/6943021 -- /home/teuthworker/archive/dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi/6943021/orig.config.yaml 2022-07-21T20:16:36.378 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-21T20:16:36.471 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi/6943021 branch: quincy description: rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} email: null first_in_suite: false job_id: '6943021' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug ms: 1 debug osd: 20 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) sha1: b6e46b8939c67a6cc754abb4d0ece3c8918eccc3 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: b6e46b8939c67a6cc754abb4d0ece3c8918eccc3 rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 workunit: branch: pacific sha1: c1e9a288b84fe43101b2ee3898d2a85405526623 owner: scheduled_dgalloway@teuthology priority: 50 repo: git@github.com:ceph/ceph-releases.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: b6e46b8939c67a6cc754abb4d0ece3c8918eccc3 sleep_before_teardown: 0 suite: rgw suite_branch: pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph_c1e9a288b84fe43101b2ee3898d2a85405526623/qa suite_relpath: qa suite_repo: git@github.com:ceph/ceph.git suite_sha1: c1e9a288b84fe43101b2ee3898d2a85405526623 targets: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkiRdg5K8JVu/cZo3BshI2ZsdhNRVRlXEly6Q9Gc4Pb2fS3VqltO6Yd6skKGTn6wFkIp4M6nT53Wn1sDimrk61sU43AaKkualpYhrErSwx2z7nwFFmLjA6+ONC6lXc0R/iSCxQddB0GlOiDbOh1F832FAlTMU08iIN/Q/BJZYlM7A1LNRJx76DGgspeCnSDW+shr4r0L5t7r0H+qXOTC61yiqwOZzpUfCvLpmmG09FrSW7YlenpNZxlX3d5a9CkO8Mr5PGipYav45pykbCp2kb4TEghgtGpC4sBrK7mEl7MP8j8wwfufkDPtZbWKyeL+WMF1J00eJHAxtyOIio8SEaMBRhgbnmGcURZ2XIN8mzb/U1jPdCY5XPQ6FkbbU7e+RFPmy/ogunJjIzpCuZ3bK0vE96bEvUV8m50ERjak7iilUeyZfL73DPdF1tBuOWjq3BGNRgi4DwDLRI2ZLWO436F3Yc66v4s50TIiWYhL/TAymKeEgDZ0KWhITukHbuRSE= smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmpWqNoyaZ07jGW6aUkiXNV1lHuYYbE6yPW50k1cDBGduiAQj3D2+cYNPyx+CoDnB2eFoWO1rxhkEDH0vvyBfaoXx8HxdqYevKJOSiUctWX9lF0M5VDHgvHlrxCrW894kVCdn1HSCP5JySn20KbNBF5lV8wQQP85OaUc6ksZOPcm3yoJA3l3VgnCYS5TGJ7KtoayPLSBuRhUjvvyaAmT8bJp5A2VTJpkKk94wBb1aacFkVzS/Y3zrCVFObIyUgucfav5uQhFgE5uwLfa6UnkGim+FE/RuSDO9ASqB8CALUcHXsnPTUKp+pnrWgaWF0Efm6EhPDC7Jah+fx3vRK1udmfXsj5uMkyLVMw4IxUUwvtAcgDsACGNg99XqhzWgVPRXQDhD8m/Z5H5cYAxMYqEYu0n4NB0r+goACFjrEMN332XHICF63De9EdmCYOxyZiYA2WCfRteOUu8IlSruOVF777CAfZGdQddLC7BrHZPNcqBgDo7dXFxNEJJ95aGZEIrU= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8d598431210977f8caccec83230b4bfec7bd5d3f timestamp: 2022-07-21_20:03:26 tube: smithi user: dgalloway verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-21T20:16:36.472 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c1e9a288b84fe43101b2ee3898d2a85405526623/qa; will attempt to use it 2022-07-21T20:16:36.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c1e9a288b84fe43101b2ee3898d2a85405526623/qa/tasks 2022-07-21T20:16:36.473 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-21T20:16:36.483 INFO:teuthology.task.internal:Checking packages... 2022-07-21T20:16:36.513 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b6e46b8939c67a6cc754abb4d0ece3c8918eccc3' 2022-07-21T20:16:36.514 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-21T20:16:36.515 INFO:teuthology.packaging:ref: None 2022-07-21T20:16:36.515 INFO:teuthology.packaging:tag: None 2022-07-21T20:16:36.516 INFO:teuthology.packaging:branch: quincy 2022-07-21T20:16:36.516 INFO:teuthology.packaging:sha1: b6e46b8939c67a6cc754abb4d0ece3c8918eccc3 2022-07-21T20:16:36.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=quincy 2022-07-21T20:16:36.768 INFO:teuthology.task.internal:Found packages for ceph version 17.2.2-1focal 2022-07-21T20:16:36.769 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-21T20:16:36.777 INFO:teuthology.task.internal:no buildpackages task found 2022-07-21T20:16:36.777 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-21T20:16:36.784 INFO:teuthology.task.internal:Saving configuration 2022-07-21T20:16:36.817 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-21T20:16:36.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-21T20:16:36.851 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi/6943021', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 20:11:14.942389', 'locked_by': 'scheduled_dgalloway@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJTWWLFc8URZubtL7OqoiHfDG2E+eLIv2Uwt6xmFSsE/'} 2022-07-21T20:16:36.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dgalloway-2022-07-21_20:03:26-rgw-quincy-distro-default-smithi/6943021', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-21 20:11:14.941015', 'locked_by': 'scheduled_dgalloway@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAIxAhMmHoFuitL8QsYZU2aIx9rRq04fROt5aLlG7Ikh'} 2022-07-21T20:16:36.880 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-21T20:16:36.904 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2022-07-21T20:16:36.904 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2022-07-21T20:16:36.905 INFO:teuthology.run_tasks:Running task console_log... 2022-07-21T20:16:37.005 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f43e2ae2e18>, signals=[15]) 2022-07-21T20:16:37.008 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-21T20:16:37.019 INFO:teuthology.task.internal:Opening connections... 2022-07-21T20:16:37.020 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2022-07-21T20:16:37.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T20:16:37.141 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2022-07-21T20:16:37.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-21T20:16:37.243 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-21T20:16:37.254 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2022-07-21T20:16:37.267 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2022-07-21T20:16:37.268 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2022-07-21T20:16:37.314 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2022-07-21T20:16:37.315 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T20:16:37.315 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2022-07-21T20:16:37.316 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2022-07-21T20:16:37.316 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T20:16:37.316 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="20.04" 2022-07-21T20:16:37.317 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T20:16:37.317 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T20:16:37.317 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T20:16:37.317 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T20:16:37.318 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=focal 2022-07-21T20:16:37.318 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=focal 2022-07-21T20:16:37.322 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2022-07-21T20:16:37.362 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2022-07-21T20:16:37.373 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2022-07-21T20:16:37.374 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2022-07-21T20:16:37.420 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="Ubuntu" 2022-07-21T20:16:37.421 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-21T20:16:37.421 INFO:teuthology.orchestra.run.smithi096.stdout:ID=ubuntu 2022-07-21T20:16:37.421 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE=debian 2022-07-21T20:16:37.421 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-21T20:16:37.422 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="20.04" 2022-07-21T20:16:37.422 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-21T20:16:37.422 INFO:teuthology.orchestra.run.smithi096.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-21T20:16:37.423 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-21T20:16:37.423 INFO:teuthology.orchestra.run.smithi096.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-21T20:16:37.423 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_CODENAME=focal 2022-07-21T20:16:37.424 INFO:teuthology.orchestra.run.smithi096.stdout:UBUNTU_CODENAME=focal 2022-07-21T20:16:37.424 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2022-07-21T20:16:37.455 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-21T20:16:37.476 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-21T20:16:37.484 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-21T20:16:37.484 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2022-07-21T20:16:37.487 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2022-07-21T20:16:37.494 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-21T20:16:37.503 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-21T20:16:37.504 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2022-07-21T20:16:37.535 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2022-07-21T20:16:37.546 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-21T20:16:37.615 INFO:teuthology.run_tasks:Running task kernel... 2022-07-21T20:16:37.644 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-21T20:16:37.645 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-21T20:16:37.646 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T20:16:37.649 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-21T20:16:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T20:16:37.655 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2022-07-21T20:16:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-21T20:16:37.657 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2022-07-21T20:16:37.664 INFO:teuthology.orchestra.run.smithi033.stdout:5.4.0-122-generic 2022-07-21T20:16:37.665 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.4.0-122-generic 2022-07-21T20:16:37.665 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2022-07-21T20:16:37.704 INFO:teuthology.orchestra.run.smithi096.stdout:5.4.0-122-generic 2022-07-21T20:16:37.705 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.4.0-122-generic 2022-07-21T20:16:37.705 DEBUG:teuthology.orchestra.run.smithi096:> sudo apt-get clean 2022-07-21T20:16:37.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo apt-get update 2022-07-21T20:16:37.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2022-07-21T20:16:37.975 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T20:16:38.083 INFO:teuthology.orchestra.run.smithi096.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T20:16:38.083 INFO:teuthology.orchestra.run.smithi096.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-21T20:16:38.092 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-21T20:16:38.147 INFO:teuthology.orchestra.run.smithi096.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T20:16:38.173 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-21T20:16:38.284 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T20:16:38.504 INFO:teuthology.orchestra.run.smithi096.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T20:16:38.513 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-21T20:16:38.609 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T20:16:38.624 INFO:teuthology.orchestra.run.smithi096.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,630 kB] 2022-07-21T20:16:38.676 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T20:16:38.718 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T20:16:38.719 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T20:16:38.720 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T20:16:38.761 INFO:teuthology.orchestra.run.smithi096.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,990 kB] 2022-07-21T20:16:38.887 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T20:16:38.923 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T20:16:39.008 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T20:16:39.113 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T20:16:39.119 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T20:16:40.917 INFO:teuthology.orchestra.run.smithi096.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-21T20:16:41.303 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T20:16:41.524 INFO:teuthology.orchestra.run.smithi096.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-21T20:16:41.537 INFO:teuthology.orchestra.run.smithi096.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [696 kB] 2022-07-21T20:16:41.911 INFO:teuthology.orchestra.run.smithi096.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-21T20:16:41.915 INFO:teuthology.orchestra.run.smithi096.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,091 kB] 2022-07-21T20:16:42.263 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T20:16:42.444 INFO:teuthology.orchestra.run.smithi096.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-21T20:16:42.746 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T20:16:42.753 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T20:16:42.933 INFO:teuthology.orchestra.run.smithi096.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-21T20:16:42.940 INFO:teuthology.orchestra.run.smithi096.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-21T20:16:43.413 INFO:teuthology.orchestra.run.smithi096.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [155 kB] 2022-07-21T20:16:43.614 INFO:teuthology.orchestra.run.smithi096.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [711 kB] 2022-07-21T20:16:44.325 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T20:16:44.529 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T20:16:44.530 INFO:teuthology.orchestra.run.smithi096.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-21T20:16:44.591 INFO:teuthology.orchestra.run.smithi096.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-21T20:16:44.735 INFO:teuthology.orchestra.run.smithi096.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T20:16:45.303 INFO:teuthology.orchestra.run.smithi096.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-21T20:16:45.410 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-21T20:16:45.492 INFO:teuthology.orchestra.run.smithi096.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-07-21T20:16:46.002 INFO:teuthology.orchestra.run.smithi096.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-21T20:16:46.683 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T20:16:46.896 INFO:teuthology.orchestra.run.smithi096.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-21T20:16:46.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T20:16:47.188 INFO:teuthology.orchestra.run.smithi096.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-21T20:16:49.350 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 11.6 MB in 9s (1,278 kB/s) 2022-07-21T20:16:49.459 INFO:teuthology.orchestra.run.smithi096.stdout:Fetched 11.6 MB in 9s (1,243 kB/s) 2022-07-21T20:16:50.690 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-07-21T20:16:50.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T20:16:50.781 INFO:teuthology.orchestra.run.smithi096.stdout:Reading package lists... 2022-07-21T20:16:50.783 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2022-07-21T20:16:50.807 DEBUG:teuthology.orchestra.run.smithi096:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-21T20:16:50.877 INFO:teuthology.orchestra.run.smithi096.stdout:Reading package lists... 2022-07-21T20:16:50.964 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2022-07-21T20:16:50.965 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2022-07-21T20:16:51.054 INFO:teuthology.orchestra.run.smithi096.stdout:Building dependency tree... 2022-07-21T20:16:51.057 INFO:teuthology.orchestra.run.smithi096.stdout:Reading state information... 2022-07-21T20:16:51.203 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T20:16:51.203 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic set to manually installed. 2022-07-21T20:16:51.204 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T20:16:51.206 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2022-07-21T20:16:51.232 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2022-07-21T20:16:51.233 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2022-07-21T20:16:51.233 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2022-07-21T20:16:51.234 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2022-07-21T20:16:51.234 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 19 2022-07-21T20:16:51.235 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T20:16:51.235 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2022-07-21T20:16:51.236 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2022-07-21T20:16:51.236 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.4.0.122.123 2022-07-21T20:16:51.237 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T20:16:51.237 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T20:16:51.238 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2022-07-21T20:16:51.238 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2022-07-21T20:16:51.239 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2022-07-21T20:16:51.240 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2022-07-21T20:16:51.241 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T20:16:51.241 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2022-07-21T20:16:51.242 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T20:16:51.264 INFO:teuthology.orchestra.run.smithi096.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-21T20:16:51.265 INFO:teuthology.orchestra.run.smithi096.stdout:linux-image-generic set to manually installed. 2022-07-21T20:16:51.265 INFO:teuthology.orchestra.run.smithi096.stdout:0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. 2022-07-21T20:16:51.267 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2022-07-21T20:16:51.268 DEBUG:teuthology.orchestra.run.smithi096:> dpkg -s linux-image-generic 2022-07-21T20:16:51.287 INFO:teuthology.orchestra.run.smithi096.stdout:Package: linux-image-generic 2022-07-21T20:16:51.287 INFO:teuthology.orchestra.run.smithi096.stdout:Status: install ok installed 2022-07-21T20:16:51.287 INFO:teuthology.orchestra.run.smithi096.stdout:Priority: optional 2022-07-21T20:16:51.288 INFO:teuthology.orchestra.run.smithi096.stdout:Section: kernel 2022-07-21T20:16:51.288 INFO:teuthology.orchestra.run.smithi096.stdout:Installed-Size: 19 2022-07-21T20:16:51.288 INFO:teuthology.orchestra.run.smithi096.stdout:Maintainer: Ubuntu Kernel Team 2022-07-21T20:16:51.288 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: amd64 2022-07-21T20:16:51.289 INFO:teuthology.orchestra.run.smithi096.stdout:Source: linux-meta 2022-07-21T20:16:51.289 INFO:teuthology.orchestra.run.smithi096.stdout:Version: 5.4.0.122.123 2022-07-21T20:16:51.289 INFO:teuthology.orchestra.run.smithi096.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-21T20:16:51.289 INFO:teuthology.orchestra.run.smithi096.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-21T20:16:51.290 INFO:teuthology.orchestra.run.smithi096.stdout:Recommends: thermald 2022-07-21T20:16:51.290 INFO:teuthology.orchestra.run.smithi096.stdout:Description: Generic Linux kernel image 2022-07-21T20:16:51.290 INFO:teuthology.orchestra.run.smithi096.stdout: This package will always depend on the latest generic kernel image 2022-07-21T20:16:51.290 INFO:teuthology.orchestra.run.smithi096.stdout: available. 2022-07-21T20:16:51.291 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-21T20:16:51.291 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2022-07-21T20:16:51.292 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-21T20:16:51.299 DEBUG:teuthology.parallel:result is None 2022-07-21T20:16:51.310 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2022-07-21T20:16:51.333 DEBUG:teuthology.parallel:result is None 2022-07-21T20:16:51.333 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-21T20:16:51.341 INFO:teuthology.task.internal:Creating test directory... 2022-07-21T20:16:51.342 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T20:16:51.345 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-21T20:16:51.381 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-21T20:16:51.389 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-21T20:16:51.397 INFO:teuthology.task.internal:Creating archive directory... 2022-07-21T20:16:51.397 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T20:16:51.401 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-21T20:16:51.414 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-21T20:16:51.421 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-21T20:16:51.422 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T20:16:51.451 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-21T20:16:51.481 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T20:16:51.483 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T20:16:51.492 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T20:16:51.494 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-21T20:16:51.496 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-21T20:16:51.509 INFO:teuthology.task.internal:Configuring sudo... 2022-07-21T20:16:51.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T20:16:51.513 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-21T20:16:51.554 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-21T20:16:51.568 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-21T20:16:51.569 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T20:16:51.572 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-21T20:16:51.580 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T20:16:51.587 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T20:16:51.640 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2022-07-21T20:16:51.641 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T20:16:51.662 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-21T20:16:51.670 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-21T20:16:51.722 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2022-07-21T20:16:51.723 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-21T20:16:51.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2022-07-21T20:16:51.745 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2022-07-21T20:16:51.794 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-21T20:16:51.808 INFO:teuthology.task.internal:Starting timer... 2022-07-21T20:16:51.808 INFO:teuthology.run_tasks:Running task pcp... 2022-07-21T20:16:51.821 INFO:teuthology.run_tasks:Running task selinux... 2022-07-21T20:16:51.830 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2022-07-21T20:16:51.830 DEBUG:teuthology.task.selinux:Excluding smithi096: OS 'ubuntu' does not support SELinux 2022-07-21T20:16:51.832 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-21T20:16:51.832 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-07-21T20:16:51.833 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-21T20:16:51.834 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-21T20:16:51.844 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-07-21T20:16:51.845 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-21T20:16:51.866 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-21T20:16:51.868 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi096.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-21T20:19:38.286 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-21T20:19:38.288 DEBUG:teuthology.task.console_log:Killing console logger for smithi033 2022-07-21T20:19:38.290 DEBUG:teuthology.task.console_log:Killing console logger for smithi096 2022-07-21T20:19:38.291 DEBUG:teuthology.exit:Finished running handlers