2023-10-05T19:23:08.747 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:23:08.747 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752 --owner scheduled_yuriw@teuthology --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752/orig.config.yaml 2023-10-05T19:23:08.786 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:23:08.868 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false job_id: '7411752' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 rgw crypt kmip ca path: /etc/ceph/kmiproot.crt rgw crypt kmip client cert: /etc/ceph/kmip-client.crt rgw crypt kmip client key: /etc/ceph/kmip-client.key rgw crypt kmip kms key template: pykmip-$keyid rgw crypt require ssl: false rgw crypt s3 kms backend: kmip setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 rgw: client.0: use-pykmip-role: client.0 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - openssl_keys: kmip-client: ca: kmiproot client: client.0 cn: rgw-client kmip-server: ca: kmiproot client: client.0 kmiproot: client: client.0 cn: kmiproot key-type: rsa:4096 - exec: client.0: - chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key - pykmip: client.0: clientca: kmiproot clientcert: kmip-client secrets: - name: pykmip-my-key-1 - name: pykmip-my-key-2 servercert: kmip-server - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:23:08.869 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:23:08.869 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:23:08.869 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:23:08.871 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:23:08.893 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:23:08.894 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:23:08.894 INFO:teuthology.packaging:ref: None 2023-10-05T19:23:08.894 INFO:teuthology.packaging:tag: None 2023-10-05T19:23:08.894 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:23:08.894 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:23:08.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:23:09.034 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:23:09.034 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:23:09.041 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:23:09.041 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:23:09.072 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:23:09.084 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:23:09.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:23:09.126 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:17:00.414337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-10-05T19:23:09.126 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:23:09.132 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-05T19:23:09.132 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:23:09.201 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd66ab07ee0>, signals=[15]) 2023-10-05T19:23:09.202 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:23:09.210 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:23:09.210 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:23:09.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:23:09.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:23:09.278 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2023-10-05T19:23:09.404 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2023-10-05T19:23:09.405 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2023-10-05T19:23:09.414 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2023-10-05T19:23:09.414 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-10-05T19:23:09.414 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2023-10-05T19:23:09.414 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2023-10-05T19:23:09.414 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="20.04" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=focal 2023-10-05T19:23:09.415 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=focal 2023-10-05T19:23:09.416 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2023-10-05T19:23:09.438 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:23:09.447 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:23:09.476 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:23:09.476 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:23:09.483 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:23:09.520 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:23:09.520 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:23:09.537 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:23:09.569 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:23:09.674 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:23:09.675 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:23:09.675 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:23:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:23:09.681 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-10-05T19:23:09.687 INFO:teuthology.orchestra.run.smithi064.stdout:5.4.0-124-generic 2023-10-05T19:23:09.687 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.4.0-124-generic 2023-10-05T19:23:09.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-10-05T19:23:09.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-05T19:23:09.997 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-10-05T19:23:10.101 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:23:10.183 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-10-05T19:23:10.285 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [659 kB] 2023-10-05T19:23:10.386 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,495 kB] 2023-10-05T19:23:10.431 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [389 kB] 2023-10-05T19:23:10.435 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-10-05T19:23:10.435 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-10-05T19:23:10.435 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,257 kB] 2023-10-05T19:23:10.455 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [316 kB] 2023-10-05T19:23:10.457 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:23:10.457 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [616 kB] 2023-10-05T19:23:10.463 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [890 kB] 2023-10-05T19:23:10.471 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB] 2023-10-05T19:23:10.472 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-10-05T19:23:10.476 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-10-05T19:23:10.477 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-10-05T19:23:10.477 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-10-05T19:23:10.477 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-10-05T19:23:10.529 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-10-05T19:23:10.670 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [893 kB] 2023-10-05T19:23:10.917 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,889 kB] 2023-10-05T19:23:11.080 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB] 2023-10-05T19:23:11.093 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-10-05T19:23:11.094 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,383 kB] 2023-10-05T19:23:11.154 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.6 kB] 2023-10-05T19:23:11.155 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [334 kB] 2023-10-05T19:23:11.175 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:23:11.339 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [749 kB] 2023-10-05T19:23:11.930 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,121 kB] 2023-10-05T19:23:12.039 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB] 2023-10-05T19:23:12.060 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-10-05T19:23:12.061 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-10-05T19:23:12.063 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-10-05T19:23:12.064 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-10-05T19:23:12.064 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-10-05T19:23:12.064 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-10-05T19:23:12.069 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-10-05T19:23:12.070 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-10-05T19:23:12.234 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-10-05T19:23:12.398 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-10-05T19:23:12.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-10-05T19:23:14.450 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 17.6 MB in 3s (6,938 kB/s) 2023-10-05T19:23:15.926 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:23:15.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:23:16.027 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:23:16.235 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:23:16.236 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:23:16.447 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-05T19:23:16.447 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:23:16.448 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-generic linux-image-5.4.0-164-generic 2023-10-05T19:23:16.448 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-5.4.0-164-generic linux-modules-extra-5.4.0-164-generic 2023-10-05T19:23:16.448 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-05T19:23:16.449 INFO:teuthology.orchestra.run.smithi064.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-10-05T19:23:16.449 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-10-05T19:23:16.449 INFO:teuthology.orchestra.run.smithi064.stdout: thermald 2023-10-05T19:23:16.512 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:23:16.512 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:23:16.512 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-164-generic linux-modules-5.4.0-164-generic 2023-10-05T19:23:16.513 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-extra-5.4.0-164-generic 2023-10-05T19:23:16.514 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-05T19:23:16.514 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-05T19:23:16.568 INFO:teuthology.orchestra.run.smithi064.stdout:3 upgraded, 5 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:23:16.568 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 77.0 MB of archives. 2023-10-05T19:23:16.568 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 380 MB of additional disk space will be used. 2023-10-05T19:23:16.568 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-164-generic amd64 5.4.0-164.181 [14.9 MB] 2023-10-05T19:23:16.896 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-164-generic amd64 5.4.0-164.181 [10.5 MB] 2023-10-05T19:23:16.988 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-164-generic amd64 5.4.0-164.181 [39.2 MB] 2023-10-05T19:23:17.510 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.164.161 [1,896 B] 2023-10-05T19:23:17.510 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.164.161 [2,592 B] 2023-10-05T19:23:17.511 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164 all 5.4.0-164.181 [11.0 MB] 2023-10-05T19:23:17.609 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164-generic amd64 5.4.0-164.181 [1,360 kB] 2023-10-05T19:23:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.164.161 [2,468 B] 2023-10-05T19:23:18.344 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 77.0 MB in 1s (70.1 MB/s) 2023-10-05T19:23:18.487 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-5.4.0-164-generic. 2023-10-05T19:23:20.196 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-10-05T19:23:20.201 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-linux-modules-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:23:20.309 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:22.276 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-image-5.4.0-164-generic. 2023-10-05T19:23:22.290 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-linux-image-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:23:22.390 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:23.513 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-164-generic. 2023-10-05T19:23:23.529 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:23:23.570 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:28.372 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-linux-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:23:28.524 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:23:28.883 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:23:29.011 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:23:29.356 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.4.0-164. 2023-10-05T19:23:29.371 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-linux-headers-5.4.0-164_5.4.0-164.181_all.deb ... 2023-10-05T19:23:29.414 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:23:35.855 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.4.0-164-generic. 2023-10-05T19:23:35.879 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-linux-headers-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:23:35.921 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:38.794 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:23:38.939 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:23:39.269 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:23:39.410 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:39.536 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:40.804 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:42.002 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:23:42.002 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-164-generic 2023-10-05T19:23:42.165 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-generic (5.4.0.164.161) ... 2023-10-05T19:23:42.290 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:44.059 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-generic (5.4.0.164.161) ... 2023-10-05T19:23:44.202 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-generic (5.4.0.164.161) ... 2023-10-05T19:23:44.328 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:23:44.407 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-05T19:23:44.408 INFO:teuthology.orchestra.run.smithi064.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:24:06.724 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-05T19:24:06.724 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub' 2023-10-05T19:24:06.730 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:24:06.750 INFO:teuthology.orchestra.run.smithi064.stdout:Generating grub configuration file ... 2023-10-05T19:24:07.073 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:24:07.093 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:24:07.495 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:24:07.507 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:24:09.287 INFO:teuthology.orchestra.run.smithi064.stdout:done 2023-10-05T19:24:09.739 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-10-05T19:24:09.809 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-10-05T19:24:09.809 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-10-05T19:24:09.809 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-10-05T19:24:09.810 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.4.0.164.161 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:24:09.811 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-10-05T19:24:09.812 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-164-generic 2023-10-05T19:24:09.812 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:24:09.812 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:24:09.812 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:24:09.813 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:24:09.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-10-05T19:24:09.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-05T19:24:10.026 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:24:10.026 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:24:10.033 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:24:10.050 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:24:11.798 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:24:11.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:24:11.893 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:24:12.100 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:24:12.101 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:24:12.334 INFO:teuthology.orchestra.run.smithi064.stdout:linux-image-generic is already the newest version (5.4.0.164.161). 2023-10-05T19:24:12.334 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:24:12.336 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:24:12.357 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.4.0.164.161 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:24:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-10-05T19:24:12.358 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-10-05T19:24:12.406 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.zSXfemNDnB 2023-10-05T19:24:12.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zSXfemNDnB 2023-10-05T19:24:12.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.zSXfemNDnB 2023-10-05T19:24:12.465 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.zSXfemNDnB is 9KB 2023-10-05T19:24:12.479 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.zSXfemNDnB 2023-10-05T19:24:12.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-05T19:24:12.539 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:24:12.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-05T19:24:12.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-05T19:24:12.590 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-164-generic 2023-10-05T19:24:12.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo update-grub 2023-10-05T19:24:13.894 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub' 2023-10-05T19:24:13.914 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:24:13.924 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2023-10-05T19:24:14.228 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:24:14.240 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:24:14.583 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:24:14.595 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:24:15.328 INFO:teuthology.orchestra.run.smithi064.stderr:done 2023-10-05T19:24:15.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2023-10-05T19:24:45.362 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:24:45.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:24:45.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:25:45.423 DEBUG:teuthology.orchestra.remote:timed out 2023-10-05T19:25:54.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:25:54.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:25:55.451 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-10-05T19:26:07.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:26:07.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:26:07.670 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-10-05T19:26:08.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:26:08.597 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-164-generic"... 2023-10-05T19:26:08.597 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-10-05T19:26:08.604 INFO:teuthology.orchestra.run.smithi064.stdout:5.4.0-164-generic 2023-10-05T19:26:08.604 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-164-generic vs 5.4.0-164-generic 2023-10-05T19:26:08.604 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:26:08.604 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-05T19:26:09.606 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T19:26:09.606 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:26:09.700 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2023-10-05T19:26:09.721 DEBUG:teuthology.parallel:result is None 2023-10-05T19:26:09.721 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T19:26:09.728 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T19:26:09.729 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:26:09.735 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T19:26:09.742 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T19:26:09.752 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T19:26:09.752 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:26:09.767 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T19:26:09.773 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T19:26:09.773 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:26:09.844 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:26:09.856 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:26:09.858 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T19:26:09.865 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T19:26:09.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:26:09.881 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T19:26:09.934 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T19:26:09.934 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:26:09.941 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:26:09.948 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:26:09.955 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:26:09.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:26:09.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2023-10-05T19:26:10.025 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T19:26:10.032 INFO:teuthology.task.internal:Starting timer... 2023-10-05T19:26:10.032 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T19:26:10.042 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T19:26:10.051 DEBUG:teuthology.task.selinux:Excluding smithi064: OS 'ubuntu' does not support SELinux 2023-10-05T19:26:10.052 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T19:26:10.052 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-05T19:26:10.052 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T19:26:10.052 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T19:26:10.061 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-05T19:26:10.061 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T19:26:10.077 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T19:26:10.078 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T19:31:53.177 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com')] 2023-10-05T19:31:53.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:31:53.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:31:53.228 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-10-05T19:31:53.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-05T19:31:53.360 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T19:31:53.371 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T19:31:53.371 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:31:53.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:31:53.404 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-10-05T19:31:53.404 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Command line: ntpd -gq 2023-10-05T19:31:53.405 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: proto: precision = 0.120 usec (-23) 2023-10-05T19:31:53.406 INFO:teuthology.orchestra.run.smithi064.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:31:53.406 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:31:53.406 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:31:53.406 INFO:teuthology.orchestra.run.smithi064.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:31:53.407 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen and drop on 0 v6wildcard [::]:123 2023-10-05T19:31:53.407 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-05T19:31:53.407 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen normally on 2 lo 127.0.0.1:123 2023-10-05T19:31:53.408 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen normally on 3 enp3s0f1 172.21.15.64:123 2023-10-05T19:31:53.408 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen normally on 4 lo [::1]:123 2023-10-05T19:31:53.408 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8df%5]:123 2023-10-05T19:31:53.408 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:31:53 ntpd[22857]: Listening on routing socket on fd #22 for interface updates 2023-10-05T19:31:54.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:54.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:54.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:54.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:54.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:54.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:54 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:56.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:56.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:56 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:57.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:57.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:57.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:57.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:57.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:57.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:57 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:58.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:58.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:58.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:58 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:31:59.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:59.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:59.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:31:59.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:31:59.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:31:59.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:31:59 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:32:00.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:32:00.405 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stdout: 5 Oct 19:32:00 ntpd[22857]: ntpd: time slew -0.001577 s 2023-10-05T19:32:00.406 INFO:teuthology.orchestra.run.smithi064.stdout:ntpd: time slew -0.001577s 2023-10-05T19:32:00.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-05T19:32:00.407 INFO:teuthology.orchestra.run.smithi064.stderr: 5 Oct 19:32:00 ntpd[22857]: can't open /var/log/ntpstats/loopstats.20231005: Permission denied 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:32:00.475 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:32:00.475 INFO:teuthology.run_tasks:Running task install... 2023-10-05T19:32:00.484 DEBUG:teuthology.task.install:project ceph 2023-10-05T19:32:00.484 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T19:32:00.484 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T19:32:00.484 INFO:teuthology.task.install:Using flavor: default 2023-10-05T19:32:00.490 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T19:32:00.490 INFO:teuthology.task.install:extra packages: [] 2023-10-05T19:32:00.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-key list | grep Ceph 2023-10-05T19:32:00.592 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-10-05T19:32:00.656 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-05T19:32:00.656 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph.com (release key) 2023-10-05T19:32:00.657 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-05T19:32:00.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:32:00.805 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default/ 2023-10-05T19:32:00.805 INFO:teuthology.task.install.deb:Package version is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:32:01.022 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:32:01.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-05T19:32:01.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-05T19:32:01.186 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:32:01.298 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:32:01.380 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:32:01.434 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal InRelease 2023-10-05T19:32:01.462 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:32:01.514 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-10-05T19:32:01.600 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release.gpg 2023-10-05T19:32:01.689 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-10-05T19:32:01.811 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main i386 Packages [3,372 B] 2023-10-05T19:32:01.908 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 28.4 kB in 1s (40.5 kB/s) 2023-10-05T19:32:02.984 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:32:03.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1278-g487ffcb5-1focal cephadm=17.2.6-1278-g487ffcb5-1focal ceph-mds=17.2.6-1278-g487ffcb5-1focal ceph-mgr=17.2.6-1278-g487ffcb5-1focal ceph-common=17.2.6-1278-g487ffcb5-1focal ceph-fuse=17.2.6-1278-g487ffcb5-1focal ceph-test=17.2.6-1278-g487ffcb5-1focal ceph-volume=17.2.6-1278-g487ffcb5-1focal radosgw=17.2.6-1278-g487ffcb5-1focal python3-rados=17.2.6-1278-g487ffcb5-1focal python3-rgw=17.2.6-1278-g487ffcb5-1focal python3-cephfs=17.2.6-1278-g487ffcb5-1focal python3-rbd=17.2.6-1278-g487ffcb5-1focal libcephfs2=17.2.6-1278-g487ffcb5-1focal libcephfs-dev=17.2.6-1278-g487ffcb5-1focal librados2=17.2.6-1278-g487ffcb5-1focal librbd1=17.2.6-1278-g487ffcb5-1focal rbd-fuse=17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:32:03.080 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:32:03.285 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:32:03.285 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:32:03.474 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-10-05T19:32:03.475 INFO:teuthology.orchestra.run.smithi064.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-10-05T19:32:03.477 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-05T19:32:03.478 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: socat xmlstarlet 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-10-05T19:32:03.479 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-10-05T19:32:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: btrfs-tools 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-10-05T19:32:03.596 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-10-05T19:32:03.598 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-10-05T19:32:03.598 INFO:teuthology.orchestra.run.smithi064.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-10-05T19:32:03.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-10-05T19:32:03.598 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-05T19:32:03.599 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-05T19:32:03.600 INFO:teuthology.orchestra.run.smithi064.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-10-05T19:32:03.600 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-05T19:32:03.600 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 librbd1 2023-10-05T19:32:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:2 upgraded, 108 newly installed, 0 to remove and 253 not upgraded. 2023-10-05T19:32:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 210 MB of archives. 2023-10-05T19:32:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 980 MB of additional disk space will be used. 2023-10-05T19:32:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-10-05T19:32:03.743 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-10-05T19:32:03.745 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-10-05T19:32:03.754 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-10-05T19:32:03.760 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-10-05T19:32:03.784 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-10-05T19:32:03.785 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-10-05T19:32:03.791 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-10-05T19:32:03.822 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-10-05T19:32:03.824 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-10-05T19:32:03.830 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-10-05T19:32:03.831 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-10-05T19:32:03.832 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-10-05T19:32:03.834 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-10-05T19:32:03.834 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-10-05T19:32:03.835 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-10-05T19:32:03.836 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-10-05T19:32:03.839 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-10-05T19:32:03.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-10-05T19:32:03.854 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-10-05T19:32:03.855 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-10-05T19:32:03.856 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-05T19:32:03.856 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-10-05T19:32:03.857 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-10-05T19:32:03.858 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-10-05T19:32:03.859 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-10-05T19:32:03.860 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-10-05T19:32:03.873 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-10-05T19:32:03.874 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-10-05T19:32:03.877 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-10-05T19:32:03.890 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-10-05T19:32:03.891 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-10-05T19:32:03.891 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-10-05T19:32:03.893 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-10-05T19:32:03.897 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-10-05T19:32:03.898 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-05T19:32:03.899 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-10-05T19:32:03.907 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-10-05T19:32:03.908 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-10-05T19:32:03.909 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-10-05T19:32:03.924 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-10-05T19:32:03.924 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-10-05T19:32:03.949 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1278-g487ffcb5-1focal [3,501 kB] 2023-10-05T19:32:04.006 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-10-05T19:32:04.022 INFO:teuthology.orchestra.run.smithi064.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-10-05T19:32:04.023 INFO:teuthology.orchestra.run.smithi064.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-10-05T19:32:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-10-05T19:32:04.094 INFO:teuthology.orchestra.run.smithi064.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-10-05T19:32:04.095 INFO:teuthology.orchestra.run.smithi064.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-10-05T19:32:04.096 INFO:teuthology.orchestra.run.smithi064.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-10-05T19:32:04.097 INFO:teuthology.orchestra.run.smithi064.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-10-05T19:32:04.098 INFO:teuthology.orchestra.run.smithi064.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-10-05T19:32:04.099 INFO:teuthology.orchestra.run.smithi064.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-10-05T19:32:04.101 INFO:teuthology.orchestra.run.smithi064.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-10-05T19:32:04.101 INFO:teuthology.orchestra.run.smithi064.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-10-05T19:32:04.114 INFO:teuthology.orchestra.run.smithi064.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-10-05T19:32:04.120 INFO:teuthology.orchestra.run.smithi064.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-10-05T19:32:04.122 INFO:teuthology.orchestra.run.smithi064.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-10-05T19:32:04.124 INFO:teuthology.orchestra.run.smithi064.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-10-05T19:32:04.125 INFO:teuthology.orchestra.run.smithi064.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-10-05T19:32:04.130 INFO:teuthology.orchestra.run.smithi064.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-10-05T19:32:04.133 INFO:teuthology.orchestra.run.smithi064.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-10-05T19:32:04.148 INFO:teuthology.orchestra.run.smithi064.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-10-05T19:32:04.151 INFO:teuthology.orchestra.run.smithi064.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-10-05T19:32:04.186 INFO:teuthology.orchestra.run.smithi064.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-10-05T19:32:04.187 INFO:teuthology.orchestra.run.smithi064.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-10-05T19:32:04.188 INFO:teuthology.orchestra.run.smithi064.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-10-05T19:32:04.188 INFO:teuthology.orchestra.run.smithi064.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-10-05T19:32:04.188 INFO:teuthology.orchestra.run.smithi064.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-10-05T19:32:04.189 INFO:teuthology.orchestra.run.smithi064.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-10-05T19:32:04.201 INFO:teuthology.orchestra.run.smithi064.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-10-05T19:32:04.201 INFO:teuthology.orchestra.run.smithi064.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-05T19:32:04.226 INFO:teuthology.orchestra.run.smithi064.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-10-05T19:32:04.226 INFO:teuthology.orchestra.run.smithi064.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-10-05T19:32:04.229 INFO:teuthology.orchestra.run.smithi064.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-10-05T19:32:04.230 INFO:teuthology.orchestra.run.smithi064.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-10-05T19:32:04.231 INFO:teuthology.orchestra.run.smithi064.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-10-05T19:32:04.231 INFO:teuthology.orchestra.run.smithi064.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-10-05T19:32:04.234 INFO:teuthology.orchestra.run.smithi064.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-10-05T19:32:04.243 INFO:teuthology.orchestra.run.smithi064.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-10-05T19:32:04.535 INFO:teuthology.orchestra.run.smithi064.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1278-g487ffcb5-1focal [3,555 kB] 2023-10-05T19:32:04.927 INFO:teuthology.orchestra.run.smithi064.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1278-g487ffcb5-1focal [760 kB] 2023-10-05T19:32:05.046 INFO:teuthology.orchestra.run.smithi064.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1278-g487ffcb5-1focal [412 kB] 2023-10-05T19:32:05.104 INFO:teuthology.orchestra.run.smithi064.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1278-g487ffcb5-1focal [22.5 kB] 2023-10-05T19:32:05.105 INFO:teuthology.orchestra.run.smithi064.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1278-g487ffcb5-1focal [200 kB] 2023-10-05T19:32:05.114 INFO:teuthology.orchestra.run.smithi064.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1278-g487ffcb5-1focal [47.2 kB] 2023-10-05T19:32:05.116 INFO:teuthology.orchestra.run.smithi064.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1278-g487ffcb5-1focal [386 kB] 2023-10-05T19:32:05.174 INFO:teuthology.orchestra.run.smithi064.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1278-g487ffcb5-1focal [5,577 kB] 2023-10-05T19:32:05.790 INFO:teuthology.orchestra.run.smithi064.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1278-g487ffcb5-1focal [116 kB] 2023-10-05T19:32:05.842 INFO:teuthology.orchestra.run.smithi064.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1278-g487ffcb5-1focal [479 kB] 2023-10-05T19:32:05.859 INFO:teuthology.orchestra.run.smithi064.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1278-g487ffcb5-1focal [27.7 MB] 2023-10-05T19:32:08.657 INFO:teuthology.orchestra.run.smithi064.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1278-g487ffcb5-1focal [6,450 kB] 2023-10-05T19:32:08.905 INFO:teuthology.orchestra.run.smithi064.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1278-g487ffcb5-1focal [221 kB] 2023-10-05T19:32:08.915 INFO:teuthology.orchestra.run.smithi064.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1278-g487ffcb5-1focal [152 kB] 2023-10-05T19:32:08.921 INFO:teuthology.orchestra.run.smithi064.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1278-g487ffcb5-1focal [1,591 kB] 2023-10-05T19:32:09.015 INFO:teuthology.orchestra.run.smithi064.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1278-g487ffcb5-1focal [7,845 kB] 2023-10-05T19:32:09.611 INFO:teuthology.orchestra.run.smithi064.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1278-g487ffcb5-1focal [28.7 MB] 2023-10-05T19:32:13.146 INFO:teuthology.orchestra.run.smithi064.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1278-g487ffcb5-1focal [3,928 B] 2023-10-05T19:32:13.216 INFO:teuthology.orchestra.run.smithi064.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1278-g487ffcb5-1focal [869 kB] 2023-10-05T19:32:13.228 INFO:teuthology.orchestra.run.smithi064.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1278-g487ffcb5-1focal [2,406 kB] 2023-10-05T19:32:13.571 INFO:teuthology.orchestra.run.smithi064.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1278-g487ffcb5-1focal [79.1 kB] 2023-10-05T19:32:13.589 INFO:teuthology.orchestra.run.smithi064.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1278-g487ffcb5-1focal [117 kB] 2023-10-05T19:32:13.594 INFO:teuthology.orchestra.run.smithi064.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1278-g487ffcb5-1focal [1,760 kB] 2023-10-05T19:32:13.852 INFO:teuthology.orchestra.run.smithi064.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1278-g487ffcb5-1focal [7,879 kB] 2023-10-05T19:32:14.952 INFO:teuthology.orchestra.run.smithi064.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1278-g487ffcb5-1focal [4,044 B] 2023-10-05T19:32:14.952 INFO:teuthology.orchestra.run.smithi064.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1278-g487ffcb5-1focal [56.3 MB] 2023-10-05T19:32:22.956 INFO:teuthology.orchestra.run.smithi064.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1278-g487ffcb5-1focal [229 kB] 2023-10-05T19:32:22.962 INFO:teuthology.orchestra.run.smithi064.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1278-g487ffcb5-1focal [21.9 kB] 2023-10-05T19:32:22.962 INFO:teuthology.orchestra.run.smithi064.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1278-g487ffcb5-1focal [17.1 MB] 2023-10-05T19:32:24.352 INFO:teuthology.orchestra.run.smithi064.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1278-g487ffcb5-1focal [79.5 kB] 2023-10-05T19:32:25.001 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 210 MB in 21s (10.1 MB/s) 2023-10-05T19:32:25.174 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-10-05T19:32:25.230 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141736 files and directories currently installed.) 2023-10-05T19:32:25.234 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-10-05T19:32:25.339 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:32:25.717 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../001-librbd1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:25.868 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librbd1 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:32:26.546 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../002-librados2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:26.655 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librados2 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:32:27.376 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs2. 2023-10-05T19:32:27.395 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:27.434 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:27.795 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rados. 2023-10-05T19:32:27.811 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:27.845 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:28.147 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-05T19:32:28.165 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:32:28.222 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:28.509 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cephfs. 2023-10-05T19:32:28.525 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:28.566 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:28.919 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-05T19:32:28.936 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:32:28.977 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:29.314 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-prettytable. 2023-10-05T19:32:29.330 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-10-05T19:32:29.398 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-10-05T19:32:29.633 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rbd. 2023-10-05T19:32:29.652 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:29.700 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:30.086 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-10-05T19:32:30.102 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-10-05T19:32:30.144 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:32:30.563 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-10-05T19:32:30.581 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-10-05T19:32:30.622 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:32:30.992 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-05T19:32:31.008 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-10-05T19:32:31.050 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:32:31.404 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-05T19:32:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-10-05T19:32:31.462 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:32:31.771 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-05T19:32:31.787 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-10-05T19:32:31.829 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:32:32.383 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-05T19:32:32.399 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:32:32.451 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:32:32.953 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-05T19:32:32.969 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:32:33.003 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:32:33.364 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-05T19:32:33.381 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:32:33.447 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:32:33.859 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-10-05T19:32:33.876 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-10-05T19:32:33.917 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:32:34.303 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librgw2. 2023-10-05T19:32:34.321 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../019-librgw2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:34.362 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:35.132 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rgw. 2023-10-05T19:32:35.149 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:35.223 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:35.617 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liboath0. 2023-10-05T19:32:35.636 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-10-05T19:32:35.675 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-10-05T19:32:36.012 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libradosstriper1. 2023-10-05T19:32:36.028 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:36.086 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:36.481 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-common. 2023-10-05T19:32:36.497 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:36.531 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:39.612 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-base. 2023-10-05T19:32:39.629 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:39.680 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:40.658 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cryptography. 2023-10-05T19:32:40.676 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-10-05T19:32:40.750 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:32:41.112 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-openssl. 2023-10-05T19:32:41.131 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-10-05T19:32:41.162 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-10-05T19:32:41.473 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-05T19:32:41.490 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:32:41.531 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:41.944 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-05T19:32:41.963 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-10-05T19:32:42.002 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:32:42.338 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-05T19:32:42.356 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-10-05T19:32:42.397 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:32:42.792 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-natsort. 2023-10-05T19:32:42.808 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-10-05T19:32:42.850 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-10-05T19:32:43.195 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jwt. 2023-10-05T19:32:43.211 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-10-05T19:32:43.253 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:32:43.590 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-logutils. 2023-10-05T19:32:43.608 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-10-05T19:32:43.639 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-10-05T19:32:43.976 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-05T19:32:43.993 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-10-05T19:32:44.034 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:32:44.362 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-mako. 2023-10-05T19:32:44.380 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-10-05T19:32:44.440 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:32:44.798 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-05T19:32:44.816 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-05T19:32:44.856 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:32:45.168 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-05T19:32:45.187 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-10-05T19:32:45.226 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:32:45.554 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webob. 2023-10-05T19:32:45.570 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-10-05T19:32:45.637 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-10-05T19:32:45.998 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-soupsieve. 2023-10-05T19:32:46.014 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-10-05T19:32:46.056 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:32:46.375 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bs4. 2023-10-05T19:32:46.395 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-10-05T19:32:46.442 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-10-05T19:32:46.854 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-waitress. 2023-10-05T19:32:46.870 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-10-05T19:32:46.915 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:32:47.273 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempita. 2023-10-05T19:32:47.289 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-10-05T19:32:47.331 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-10-05T19:32:47.701 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-paste. 2023-10-05T19:32:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-10-05T19:32:47.759 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-05T19:32:48.131 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-10-05T19:32:48.170 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:32:48.557 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-05T19:32:48.576 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-10-05T19:32:48.615 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:32:48.951 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webtest. 2023-10-05T19:32:48.968 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-10-05T19:32:49.009 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-10-05T19:32:49.354 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pecan. 2023-10-05T19:32:49.370 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-10-05T19:32:49.429 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:32:49.800 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-05T19:32:49.817 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-10-05T19:32:49.858 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:32:50.245 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-05T19:32:50.262 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:50.328 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:50.665 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr. 2023-10-05T19:32:50.684 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:50.714 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:51.276 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mon. 2023-10-05T19:32:51.293 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:51.334 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:52.513 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-osd. 2023-10-05T19:32:52.530 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:52.571 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:55.451 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph. 2023-10-05T19:32:55.471 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../052-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:55.515 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:55.852 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-fuse. 2023-10-05T19:32:55.869 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:55.901 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:56.380 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mds. 2023-10-05T19:32:56.396 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:56.437 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:56.999 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package cephadm. 2023-10-05T19:32:57.015 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../055-cephadm_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:32:57.057 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:57.402 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jinja2. 2023-10-05T19:32:57.419 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-10-05T19:32:57.460 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-10-05T19:32:57.854 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-05T19:32:57.870 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:32:57.912 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:32:58.290 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-05T19:32:58.309 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-10-05T19:32:58.425 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-05T19:32:58.712 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-underscore. 2023-10-05T19:32:58.731 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-10-05T19:32:58.776 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:32:59.079 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-10-05T19:32:59.098 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-10-05T19:32:59.137 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:32:59.489 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-routes. 2023-10-05T19:32:59.508 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-10-05T19:32:59.547 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-10-05T19:32:59.909 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-05T19:32:59.928 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:32:59.966 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:00.539 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-decorator. 2023-10-05T19:33:00.559 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-10-05T19:33:00.597 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:33:00.933 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-10-05T19:33:00.950 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-10-05T19:33:00.991 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:33:01.371 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-scipy. 2023-10-05T19:33:01.391 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-10-05T19:33:01.435 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-10-05T19:33:02.728 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-10-05T19:33:02.748 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-10-05T19:33:02.786 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:33:03.306 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-joblib. 2023-10-05T19:33:03.325 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-10-05T19:33:03.356 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-10-05T19:33:03.843 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn. 2023-10-05T19:33:03.863 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-10-05T19:33:03.901 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:33:04.668 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-05T19:33:04.686 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:33:04.734 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:05.954 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cachetools. 2023-10-05T19:33:05.970 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-10-05T19:33:06.012 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-10-05T19:33:06.340 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-05T19:33:06.357 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-10-05T19:33:06.398 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:33:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-05T19:33:06.785 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-10-05T19:33:06.826 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:33:07.171 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rsa. 2023-10-05T19:33:07.189 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-10-05T19:33:07.229 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rsa (4.0-3) ... 2023-10-05T19:33:07.565 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-google-auth. 2023-10-05T19:33:07.582 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-10-05T19:33:07.640 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-10-05T19:33:07.976 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-blinker. 2023-10-05T19:33:07.993 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-10-05T19:33:08.034 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:33:08.354 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-oauthlib. 2023-10-05T19:33:08.370 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-10-05T19:33:08.412 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:33:08.782 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-05T19:33:08.801 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-10-05T19:33:08.840 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:33:09.176 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-websocket. 2023-10-05T19:33:09.193 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-10-05T19:33:09.234 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:33:09.546 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-05T19:33:09.564 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-10-05T19:33:09.624 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:33:10.208 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-05T19:33:10.228 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:33:10.275 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:10.731 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-05T19:33:10.747 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-10-05T19:33:10.805 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:33:11.184 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-05T19:33:11.201 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:33:11.242 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:33:11.536 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package jq. 2023-10-05T19:33:11.553 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:33:11.594 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:33:11.872 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package socat. 2023-10-05T19:33:11.889 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-10-05T19:33:11.930 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking socat (1.7.3.3-2) ... 2023-10-05T19:33:12.276 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package xmlstarlet. 2023-10-05T19:33:12.292 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-10-05T19:33:12.334 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:33:12.705 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-test. 2023-10-05T19:33:12.723 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:33:12.763 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:18.159 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-volume. 2023-10-05T19:33:18.177 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:33:18.236 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:18.570 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package fonts-mathjax. 2023-10-05T19:33:18.589 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:33:18.628 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:33:19.199 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-05T19:33:19.216 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:33:19.257 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:19.668 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package nvme-cli. 2023-10-05T19:33:19.685 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-10-05T19:33:19.769 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:33:20.106 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjs-mathjax. 2023-10-05T19:33:20.125 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:33:20.164 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:33:21.939 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-natsort-doc. 2023-10-05T19:33:21.957 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-10-05T19:33:21.997 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-10-05T19:33:22.359 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-atomicwrites. 2023-10-05T19:33:22.380 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-10-05T19:33:22.442 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:33:22.761 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-attr. 2023-10-05T19:33:22.781 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-10-05T19:33:22.819 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-10-05T19:33:23.164 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webencodings. 2023-10-05T19:33:23.184 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-10-05T19:33:23.222 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:33:23.566 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-html5lib. 2023-10-05T19:33:23.585 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-10-05T19:33:23.649 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-10-05T19:33:24.196 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-10-05T19:33:24.216 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-10-05T19:33:24.252 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:33:24.789 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-packaging. 2023-10-05T19:33:24.806 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-10-05T19:33:24.847 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-packaging (20.3-1) ... 2023-10-05T19:33:25.158 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastescript. 2023-10-05T19:33:25.175 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-10-05T19:33:25.208 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-05T19:33:25.545 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pluggy. 2023-10-05T19:33:25.562 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-10-05T19:33:25.621 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-10-05T19:33:25.965 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-psutil. 2023-10-05T19:33:25.982 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-10-05T19:33:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:33:26.384 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-py. 2023-10-05T19:33:26.402 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-10-05T19:33:26.442 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:33:26.811 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-05T19:33:26.830 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-10-05T19:33:26.869 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:33:27.214 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-05T19:33:27.233 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-10-05T19:33:27.272 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:33:27.601 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pytest. 2023-10-05T19:33:27.621 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-10-05T19:33:27.684 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-10-05T19:33:28.054 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplejson. 2023-10-05T19:33:28.072 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-10-05T19:33:28.120 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:33:28.425 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-05T19:33:28.442 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-10-05T19:33:28.481 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:33:29.238 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package radosgw. 2023-10-05T19:33:29.256 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../108-radosgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:33:29.288 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:31.033 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-fuse. 2023-10-05T19:33:31.050 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:33:31.082 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:31.410 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-attr (19.3.0-2) ... 2023-10-05T19:33:31.708 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:33:31.859 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:33:31.985 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:33:32.211 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:32.865 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user cephadm....done 2023-10-05T19:33:33.051 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:33:33.177 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:33:33.605 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-05T19:33:33.931 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:33:34.057 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-10-05T19:33:34.208 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:33:34.526 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-10-05T19:33:34.932 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:33:35.241 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-10-05T19:33:35.560 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:33:35.928 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:33:36.237 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:36.539 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:33:36.664 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:33:36.940 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:33:37.219 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:33:37.344 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:33:37.746 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-10-05T19:33:38.038 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:33:38.323 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:33:38.674 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:33:38.958 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-10-05T19:33:39.284 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-10-05T19:33:39.611 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-packaging (20.3-1) ... 2023-10-05T19:33:39.920 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:33:40.079 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:33:40.205 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:33:40.556 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:33:40.958 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:33:41.318 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:33:41.728 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up socat (1.7.3.3-2) ... 2023-10-05T19:33:41.854 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:42.180 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:33:42.348 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-10-05T19:33:42.699 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:33:42.848 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:33:42.969 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-10-05T19:33:43.253 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:33:43.573 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:33:43.732 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:33:43.859 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:33:44.186 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:33:44.503 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-10-05T19:33:44.829 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:33:44.989 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:33:45.273 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-10-05T19:33:45.569 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:33:45.870 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:33:46.291 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-10-05T19:33:46.606 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:33:46.732 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:33:47.050 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:33:47.409 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:33:47.554 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:33:47.672 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-10-05T19:33:48.073 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:33:48.366 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librados2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:48.492 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-10-05T19:33:49.998 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:50.157 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:33:50.283 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-10-05T19:33:50.693 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-10-05T19:33:51.002 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:33:51.295 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:33:51.421 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-10-05T19:33:51.814 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:51.940 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:33:52.283 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:52.434 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:33:52.769 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:33:53.841 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librbd1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:53.967 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:33:54.118 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-10-05T19:33:54.252 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:33:54.395 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rsa (4.0-3) ... 2023-10-05T19:33:54.696 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:33:54.797 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-10-05T19:33:55.098 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:55.257 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:55.484 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:33:55.484 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:33:55.934 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:56.170 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:33:56.463 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-05T19:33:56.756 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:33:57.092 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:57.218 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:57.378 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:57.503 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:57.654 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-routes (2.4.1-2) ... 2023-10-05T19:33:57.947 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-10-05T19:33:58.241 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:58.350 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:33:58.517 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:33:59.471 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:00.023 INFO:teuthology.orchestra.run.smithi064.stdout:Adding group ceph....done 2023-10-05T19:34:00.434 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user ceph....done 2023-10-05T19:34:00.636 INFO:teuthology.orchestra.run.smithi064.stdout:Setting system user ceph properties....done 2023-10-05T19:34:00.654 INFO:teuthology.orchestra.run.smithi064.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-05T19:34:00.806 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-05T19:34:01.129 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-05T19:34:01.598 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:01.740 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:02.009 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:34:02.009 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:34:02.667 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:03.095 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-05T19:34:03.545 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:03.859 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:34:03.859 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:34:04.332 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:34:04.567 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:34:05.042 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:05.801 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:34:05.801 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:34:06.278 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:06.429 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:06.563 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:06.830 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:34:06.830 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:34:07.273 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:07.415 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:07.575 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:07.717 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:34:07.878 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:34:08.030 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-05T19:34:08.375 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:34:11.295 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-10-05T19:34:11.908 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:34:11.910 DEBUG:teuthology.parallel:result is None 2023-10-05T19:34:11.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:34:12.135 DEBUG:teuthology.orchestra.run.smithi064:> dpkg-query -W -f '${Version}' ceph 2023-10-05T19:34:12.159 INFO:teuthology.orchestra.run.smithi064.stdout:17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:34:12.160 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:34:12.160 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278-g487ffcb5-1focal is installed. 2023-10-05T19:34:12.160 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T19:34:12.160 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:12.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T19:34:12.170 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T19:34:12.170 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:12.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T19:34:12.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T19:34:12.239 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T19:34:12.239 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:12.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T19:34:12.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T19:34:12.264 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T19:34:12.264 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:12.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T19:34:12.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T19:34:12.331 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T19:34:12.406 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T19:34:12.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2023-10-05T19:34:12.423 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T19:34:12.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T19:34:12.478 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T19:34:12.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T19:34:12.501 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T19:34:12.502 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T19:34:12.502 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411752', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'rgw': {'client.0': {'use-pykmip-role': 'client.0'}}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': {'kmip-client': {'ca': 'kmiproot', 'client': 'client.0', 'cn': 'rgw-client'}, 'kmip-server': {'ca': 'kmiproot', 'client': 'client.0'}, 'kmiproot': {'client': 'client.0', 'cn': 'kmiproot', 'key-type': 'rsa:4096'}}}, {'exec': {'client.0': ['chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key']}}, {'pykmip': {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T19:34:12.503 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T19:34:12.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T19:34:12.569 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:12.570 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T19:34:12.575 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T19:34:12.575 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2023-10-05T19:34:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T19:34:12.582 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:34:12.582 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-10-05T19:34:12.583 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:34:12.583 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-05 19:30:23.838106716 +0000 2023-10-05T19:34:12.583 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-05 19:30:23.834106808 +0000 2023-10-05T19:34:12.583 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-05 19:30:23.834106808 +0000 2023-10-05T19:34:12.583 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-05T19:34:12.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T19:34:12.638 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-05T19:34:12.638 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-05T19:34:12.638 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000421461 s, 1.2 MB/s 2023-10-05T19:34:12.639 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T19:34:12.648 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-05 19:30:24.638088312 +0000 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-05 19:30:24.342095122 +0000 2023-10-05T19:34:12.654 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-05 19:30:24.342095122 +0000 2023-10-05T19:34:12.655 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-05T19:34:12.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T19:34:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-05T19:34:12.668 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-05T19:34:12.669 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000378869 s, 1.4 MB/s 2023-10-05T19:34:12.670 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T19:34:12.722 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2023-10-05T19:34:12.729 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-05 19:30:25.202075336 +0000 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-05 19:30:24.906082145 +0000 2023-10-05T19:34:12.730 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-05 19:30:24.906082145 +0000 2023-10-05T19:34:12.731 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-05T19:34:12.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T19:34:12.741 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-05T19:34:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-05T19:34:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000405385 s, 1.3 MB/s 2023-10-05T19:34:12.743 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T19:34:12.794 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-05 19:30:25.730063187 +0000 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-05 19:30:25.430070090 +0000 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-05 19:30:25.430070090 +0000 2023-10-05T19:34:12.802 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-05T19:34:12.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T19:34:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-05T19:34:12.813 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-05T19:34:12.814 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000417152 s, 1.2 MB/s 2023-10-05T19:34:12.814 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T19:34:12.866 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:34:12.867 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:34:12.867 INFO:tasks.ceph:Generating config... 2023-10-05T19:34:12.869 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T19:34:12.869 INFO:tasks.ceph:[client] rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-10-05T19:34:12.869 INFO:tasks.ceph:[client] rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-10-05T19:34:12.869 INFO:tasks.ceph:[client] rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-10-05T19:34:12.870 INFO:tasks.ceph:[client] rgw crypt kmip kms key template = pykmip-$keyid 2023-10-05T19:34:12.870 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-05T19:34:12.870 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = kmip 2023-10-05T19:34:12.870 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T19:34:12.870 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T19:34:12.870 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T19:34:12.871 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T19:34:12.871 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T19:34:12.871 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T19:34:12.871 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T19:34:12.871 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T19:34:12.872 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T19:34:12.872 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-05T19:34:12.872 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T19:34:12.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T19:34:12.909 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T19:34:12.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T19:34:12.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:34:13.007 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64')] 2023-10-05T19:34:13.007 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T19:34:13.008 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:13.008 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:34:13.013 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:13.042 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:34:13.042 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid c440a5a2-ff86-49f1-ad52-251a03eaf475 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = octopus 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:fsid c440a5a2-ff86-49f1-ad52-251a03eaf475 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2023-10-05T19:34:13.040435+0000 2023-10-05T19:34:13.043 INFO:teuthology.orchestra.run.smithi064.stdout:created 2023-10-05T19:34:13.040435+0000 2023-10-05T19:34:13.044 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 15 (octopus) 2023-10-05T19:34:13.044 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2023-10-05T19:34:13.044 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2023-10-05T19:34:13.044 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T19:34:13.045 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:34:13.050 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c440a5a2-ff86-49f1-ad52-251a03eaf475... 2023-10-05T19:34:13.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T19:34:13.130 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2023-10-05T19:34:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2023-10-05T19:34:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T19:34:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2023-10-05T19:34:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2023-10-05T19:34:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = 172.21.15.64 2023-10-05T19:34:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = c440a5a2-ff86-49f1-ad52-251a03eaf475 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2023-10-05T19:34:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2023-10-05T19:34:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-10-05T19:34:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2023-10-05T19:34:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T19:34:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2023-10-05T19:34:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: debug rgw = 20 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt kmip kms key template = pykmip-$keyid 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt require ssl = False 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: rgw crypt s3 kms backend = kmip 2023-10-05T19:34:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: setgroup = ceph 2023-10-05T19:34:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: setuser = ceph 2023-10-05T19:34:13.143 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2023-10-05T19:34:13.144 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T19:34:13.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T19:34:13.191 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T19:34:13.191 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:13.191 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T19:34:13.198 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:13.199 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T19:34:13.248 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:34:13.248 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:13.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T19:34:13.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:34:13.273 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:13.274 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:13.280 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T19:34:13.280 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T19:34:13.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:34:13.365 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:34:13.368 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T19:34:13.368 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T19:34:13.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T19:34:13.402 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T19:34:13.414 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T19:34:13.414 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:34:13.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T19:34:13.424 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:34:13.424 INFO:tasks.ceph:role: osd.0 2023-10-05T19:34:13.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:34:13.424 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T19:34:13.953 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:34:13.954 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:34:13.955 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-05T19:34:13.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T19:34:14.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T19:34:14.045 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:34:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:34:14.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T19:34:14.058 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:34:14.058 INFO:tasks.ceph:role: osd.1 2023-10-05T19:34:14.058 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:34:14.058 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T19:34:14.628 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:34:14.628 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:34:14.629 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:34:14.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-05T19:34:14.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-05T19:34:14.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T19:34:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:34:14.676 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:34:14.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T19:34:14.734 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:34:14.734 INFO:tasks.ceph:role: osd.2 2023-10-05T19:34:14.734 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:34:14.734 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T19:34:15.196 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:34:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:34:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:34:15.196 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:34:15.197 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:34:15.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-05T19:34:15.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-05T19:34:15.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T19:34:15.267 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:34:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:34:15.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.339+0000 7f091640c3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.339+0000 7f091640c3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.339+0000 7f091640c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.339+0000 7f091640c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.339+0000 7f091640c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:34:15.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.343+0000 7f091640c3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T19:34:15.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:15.911+0000 7f091640c3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:16.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:16.699+0000 7f091640c3c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:17.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T19:34:17.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T19:34:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T19:34:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:34:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:34:17.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:34:17.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:17.611+0000 7f7da5d043c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T19:34:18.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:18.167+0000 7f7da5d043c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:18.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:18.951+0000 7f7da5d043c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:19.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T19:34:19.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:19.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.891+0000 7fd0867b83c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T19:34:19.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.895+0000 7fd0867b83c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T19:34:19.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.895+0000 7fd0867b83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:34:19.899 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.895+0000 7fd0867b83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:34:19.899 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.895+0000 7fd0867b83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:34:19.899 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:19.895+0000 7fd0867b83c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T19:34:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:20.447+0000 7fd0867b83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:21.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:34:21.235+0000 7fd0867b83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:34:22.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T19:34:22.110 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T19:34:22.110 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T19:34:22.129 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T19:34:22.183 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T19:34:22.198 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.199 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T19:34:22.209 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.209 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T19:34:22.214 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T19:34:22.214 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC1Dx9lVdymFRAAHR9j5UytTZAlXP68g+oVHw== 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC3Dx9l/tt0FBAA1HhSQsbX7Gn/DySC9Z5seA== 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC5Dx9lgjuFJBAAka7+rUm0TeTZM88xDg6yrw== 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2023-10-05T19:34:22.270 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC7Dx9lnmpgNRAAB8Txs0vzcBRA+Qgj2ghp3Q== 2023-10-05T19:34:22.271 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2023-10-05T19:34:22.271 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC1Dx9l/ITVFxAA9OGE3C8cElC14O56FIzd5Q== 2023-10-05T19:34:22.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T19:34:22.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:34:22.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:34:22.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:34:22.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T19:34:22.528 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T19:34:22.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T19:34:22.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T19:34:22.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T19:34:22.611 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:34:22.617 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T19:34:22.617 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T19:34:22.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T19:34:22.620 INFO:tasks.ceph.mon.a:Started 2023-10-05T19:34:22.620 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T19:34:22.620 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T19:34:22.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T19:34:22.622 INFO:tasks.ceph.mgr.x:Started 2023-10-05T19:34:22.622 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T19:34:22.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T19:34:22.868 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2023-10-05T19:34:22.878 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T19:34:22.878 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T19:34:22.889 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T19:34:22.903 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:22.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T19:34:22.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new b96e8ea8-4da5-46ed-b092-decf44b64b9b 0 2023-10-05T19:34:22.956 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:22.951+0000 7f5a38f63000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T19:34:23.154 INFO:teuthology.orchestra.run.smithi064.stdout:0 2023-10-05T19:34:23.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 4bfb468e-1104-450b-8b9c-be1b059ffc68 1 2023-10-05T19:34:23.362 INFO:teuthology.orchestra.run.smithi064.stdout:1 2023-10-05T19:34:23.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 2128b6c4-0dba-4886-9d99-0513312ced44 2 2023-10-05T19:34:23.575 INFO:teuthology.orchestra.run.smithi064.stdout:2 2023-10-05T19:34:23.588 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T19:34:23.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T19:34:23.590 INFO:tasks.ceph.osd.0:Started 2023-10-05T19:34:23.590 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T19:34:23.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T19:34:23.592 INFO:tasks.ceph.osd.1:Started 2023-10-05T19:34:23.593 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T19:34:23.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T19:34:23.594 INFO:tasks.ceph.osd.2:Started 2023-10-05T19:34:23.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:23.610 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:23.607+0000 7f5a38f63000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T19:34:23.686 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:23.683+0000 7f5a38f63000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T19:34:23.798 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:23.798 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:23.569516+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:23.814 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T19:34:23.814 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T19:34:23.833 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:23.827+0000 7f5a38f63000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:34:23.948 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-05T19:34:23.943+0000 7f6e2a49a3c0 -1 Falling back to public interface 2023-10-05T19:34:24.077 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.071+0000 7f5a38f63000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T19:34:24.117 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:24.132 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.127+0000 7f5a38f63000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T19:34:24.215 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-10-05T19:34:24.211+0000 7f62da3383c0 -1 Falling back to public interface 2023-10-05T19:34:24.231 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-10-05T19:34:24.227+0000 7f66d7dfd3c0 -1 Falling back to public interface 2023-10-05T19:34:24.288 INFO:teuthology.misc.health.smithi064.stdout: 2023-10-05T19:34:24.288 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":5,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:23.569516+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:24.299 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-05T19:34:24.325 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.318+0000 7f5a38f63000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T19:34:24.394 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.390+0000 7f5a38f63000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T19:34:24.462 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.458+0000 7f5a38f63000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T19:34:24.903 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:24.898+0000 7f5a38f63000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T19:34:25.029 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.022+0000 7f5a38f63000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T19:34:25.356 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.350+0000 7f5a38f63000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T19:34:25.523 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.518+0000 7f5a38f63000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:34:25.591 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.586+0000 7f5a38f63000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T19:34:25.645 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.638+0000 7f5a38f63000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T19:34:25.705 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.698+0000 7f5a38f63000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T19:34:26.002 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:25.998+0000 7f5a38f63000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T19:34:26.060 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:26.054+0000 7f5a38f63000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T19:34:26.128 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:26.122+0000 7f5a38f63000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T19:34:26.294 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:26.290+0000 7f5a38f63000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T19:34:26.364 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:26.358+0000 7f5a38f63000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T19:34:27.108 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:27.102+0000 7f5a38f63000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T19:34:27.116 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-05T19:34:27.110+0000 7f6e2a49a3c0 -1 osd.2 0 log_to_monitors true 2023-10-05T19:34:27.272 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-10-05T19:34:27.266+0000 7f62da3383c0 -1 osd.1 0 log_to_monitors true 2023-10-05T19:34:27.296 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-10-05T19:34:27.290+0000 7f66d7dfd3c0 -1 osd.0 0 log_to_monitors true 2023-10-05T19:34:28.142 INFO:tasks.ceph.mgr.x.smithi064.stderr:2023-10-05T19:34:28.138+0000 7f5a2205b700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T19:34:29.003 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-05T19:34:28.998+0000 7f6e2745f700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T19:34:29.007 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-05T19:34:29.002+0000 7f6e1e9de700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:34:30.335 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-10-05T19:34:30.330+0000 7f62d5ad8700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T19:34:30.339 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-10-05T19:34:30.334+0000 7f62ce87c700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:34:30.357 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-10-05T19:34:30.350+0000 7f66d359d700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T19:34:30.361 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-10-05T19:34:30.354+0000 7f66cc341700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:34:30.602 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:30.886 INFO:teuthology.misc.health.smithi064.stdout: 2023-10-05T19:34:30.886 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":8,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:29.170059+0000","last_up_change":"2023-10-05T19:34:29.170059+0000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":29054},{"type":"v1","addr":"172.21.15.64:6801","nonce":29054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":29054},{"type":"v1","addr":"172.21.15.64:6803","nonce":29054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":29054},{"type":"v1","addr":"172.21.15.64:6807","nonce":29054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":29054},{"type":"v1","addr":"172.21.15.64:6805","nonce":29054}]},"public_addr":"172.21.15.64:6801/29054","cluster_addr":"172.21.15.64:6803/29054","heartbeat_back_addr":"172.21.15.64:6807/29054","heartbeat_front_addr":"172.21.15.64:6805/29054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:30.899 DEBUG:teuthology.misc:1 of 3 OSDs are up 2023-10-05T19:34:37.202 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:37.485 INFO:teuthology.misc.health.smithi064.stdout: 2023-10-05T19:34:37.485 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":12,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:34.018921+0000","last_up_change":"2023-10-05T19:34:31.007994+0000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:34:31.195305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":29056},{"type":"v1","addr":"172.21.15.64:6817","nonce":29056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":29056},{"type":"v1","addr":"172.21.15.64:6819","nonce":29056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":29056},{"type":"v1","addr":"172.21.15.64:6823","nonce":29056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":29056},{"type":"v1","addr":"172.21.15.64:6821","nonce":29056}]},"public_addr":"172.21.15.64:6817/29056","cluster_addr":"172.21.15.64:6819/29056","heartbeat_back_addr":"172.21.15.64:6823/29056","heartbeat_front_addr":"172.21.15.64:6821/29056","state":["exists","up"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":29055},{"type":"v1","addr":"172.21.15.64:6809","nonce":29055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":29055},{"type":"v1","addr":"172.21.15.64:6811","nonce":29055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":29055},{"type":"v1","addr":"172.21.15.64:6815","nonce":29055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":29055},{"type":"v1","addr":"172.21.15.64:6813","nonce":29055}]},"public_addr":"172.21.15.64:6809/29055","cluster_addr":"172.21.15.64:6811/29055","heartbeat_back_addr":"172.21.15.64:6815/29055","heartbeat_front_addr":"172.21.15.64:6813/29055","state":["exists","up"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":29054},{"type":"v1","addr":"172.21.15.64:6801","nonce":29054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":29054},{"type":"v1","addr":"172.21.15.64:6803","nonce":29054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":29054},{"type":"v1","addr":"172.21.15.64:6807","nonce":29054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":29054},{"type":"v1","addr":"172.21.15.64:6805","nonce":29054}]},"public_addr":"172.21.15.64:6801/29054","cluster_addr":"172.21.15.64:6803/29054","heartbeat_back_addr":"172.21.15.64:6807/29054","heartbeat_front_addr":"172.21.15.64:6805/29054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.314369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.274631+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.128680+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:37.497 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-05T19:34:37.497 INFO:tasks.ceph:Creating RBD pool 2023-10-05T19:34:37.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T19:34:38.733 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2023-10-05T19:34:38.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T19:34:39.745 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T19:34:39.757 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T19:34:39.758 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T19:34:39.758 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T19:34:39.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T19:34:39.758 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T19:34:40.083 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:40.096 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":28871},{"type":"v1","addr":"172.21.15.64:6825","nonce":28871}]},"active_addr":"172.21.15.64:6825/28871","active_change":"2023-10-05T19:34:27.124982+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":556374678}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3859654125}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3424793120}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2735582371}]}]} 2023-10-05T19:34:40.098 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T19:34:40.098 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T19:34:40.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:40.383 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:39.716031+0000","last_up_change":"2023-10-05T19:34:31.007994+0000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:34:31.195305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:34:37.822834+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":29056},{"type":"v1","addr":"172.21.15.64:6817","nonce":29056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":29056},{"type":"v1","addr":"172.21.15.64:6819","nonce":29056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":29056},{"type":"v1","addr":"172.21.15.64:6823","nonce":29056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":29056},{"type":"v1","addr":"172.21.15.64:6821","nonce":29056}]},"public_addr":"172.21.15.64:6817/29056","cluster_addr":"172.21.15.64:6819/29056","heartbeat_back_addr":"172.21.15.64:6823/29056","heartbeat_front_addr":"172.21.15.64:6821/29056","state":["exists","up"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":29055},{"type":"v1","addr":"172.21.15.64:6809","nonce":29055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":29055},{"type":"v1","addr":"172.21.15.64:6811","nonce":29055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":29055},{"type":"v1","addr":"172.21.15.64:6815","nonce":29055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":29055},{"type":"v1","addr":"172.21.15.64:6813","nonce":29055}]},"public_addr":"172.21.15.64:6809/29055","cluster_addr":"172.21.15.64:6811/29055","heartbeat_back_addr":"172.21.15.64:6815/29055","heartbeat_front_addr":"172.21.15.64:6813/29055","state":["exists","up"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":29054},{"type":"v1","addr":"172.21.15.64:6801","nonce":29054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":29054},{"type":"v1","addr":"172.21.15.64:6803","nonce":29054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":29054},{"type":"v1","addr":"172.21.15.64:6807","nonce":29054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":29054},{"type":"v1","addr":"172.21.15.64:6805","nonce":29054}]},"public_addr":"172.21.15.64:6801/29054","cluster_addr":"172.21.15.64:6803/29054","heartbeat_back_addr":"172.21.15.64:6807/29054","heartbeat_front_addr":"172.21.15.64:6805/29054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.314369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.274631+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.128680+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:40.396 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T19:34:40.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:34:40.685 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:40.688 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:34:39.716031+0000","last_up_change":"2023-10-05T19:34:31.007994+0000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:34:31.195305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:34:37.822834+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":29056},{"type":"v1","addr":"172.21.15.64:6817","nonce":29056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":29056},{"type":"v1","addr":"172.21.15.64:6819","nonce":29056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":29056},{"type":"v1","addr":"172.21.15.64:6823","nonce":29056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":29056},{"type":"v1","addr":"172.21.15.64:6821","nonce":29056}]},"public_addr":"172.21.15.64:6817/29056","cluster_addr":"172.21.15.64:6819/29056","heartbeat_back_addr":"172.21.15.64:6823/29056","heartbeat_front_addr":"172.21.15.64:6821/29056","state":["exists","up"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":29055},{"type":"v1","addr":"172.21.15.64:6809","nonce":29055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":29055},{"type":"v1","addr":"172.21.15.64:6811","nonce":29055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":29055},{"type":"v1","addr":"172.21.15.64:6815","nonce":29055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":29055},{"type":"v1","addr":"172.21.15.64:6813","nonce":29055}]},"public_addr":"172.21.15.64:6809/29055","cluster_addr":"172.21.15.64:6811/29055","heartbeat_back_addr":"172.21.15.64:6815/29055","heartbeat_front_addr":"172.21.15.64:6813/29055","state":["exists","up"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":29054},{"type":"v1","addr":"172.21.15.64:6801","nonce":29054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":29054},{"type":"v1","addr":"172.21.15.64:6803","nonce":29054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":29054},{"type":"v1","addr":"172.21.15.64:6807","nonce":29054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":29054},{"type":"v1","addr":"172.21.15.64:6805","nonce":29054}]},"public_addr":"172.21.15.64:6801/29054","cluster_addr":"172.21.15.64:6803/29054","heartbeat_back_addr":"172.21.15.64:6807/29054","heartbeat_front_addr":"172.21.15.64:6805/29054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.314369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.274631+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.128680+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:34:40.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T19:34:40.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T19:34:40.706 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T19:34:40.850 INFO:teuthology.orchestra.run.smithi064.stdout:38654705668 2023-10-05T19:34:40.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:34:40.854 INFO:teuthology.orchestra.run.smithi064.stdout:38654705669 2023-10-05T19:34:40.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:34:40.872 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2023-10-05T19:34:40.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:34:41.122 INFO:teuthology.orchestra.run.smithi064.stdout:38654705668 2023-10-05T19:34:41.140 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2023-10-05T19:34:41.147 INFO:teuthology.orchestra.run.smithi064.stdout:38654705667 2023-10-05T19:34:41.153 INFO:teuthology.orchestra.run.smithi064.stdout:34359738370 2023-10-05T19:34:41.160 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2023-10-05T19:34:41.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.2 2023-10-05T19:34:42.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:34:42.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:34:42.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:34:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:38654705669 2023-10-05T19:34:42.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2023-10-05T19:34:42.442 DEBUG:teuthology.parallel:result is None 2023-10-05T19:34:42.445 INFO:teuthology.orchestra.run.smithi064.stdout:34359738372 2023-10-05T19:34:42.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-10-05T19:34:42.459 INFO:teuthology.orchestra.run.smithi064.stdout:38654705668 2023-10-05T19:34:42.459 DEBUG:teuthology.parallel:result is None 2023-10-05T19:34:42.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2023-10-05T19:34:42.471 DEBUG:teuthology.parallel:result is None 2023-10-05T19:34:42.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T19:34:42.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:34:42.747 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:42.750 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-10-05T19:34:42.760 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-05T19:34:41.135644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893384,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679416,"statfs":{"total":322122547200,"available":321207721984,"internally_reserved":0,"allocated":1556480,"data_stored":1281185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744802+0000","last_change":"2023-10-05T19:34:39.744802+0000","last_active":"2023-10-05T19:34:39.744802+0000","last_peered":"2023-10-05T19:34:39.744802+0000","last_clean":"2023-10-05T19:34:39.744802+0000","last_became_active":"2023-10-05T19:34:39.744411+0000","last_became_peered":"2023-10-05T19:34:39.744411+0000","last_unstale":"2023-10-05T19:34:39.744802+0000","last_undegraded":"2023-10-05T19:34:39.744802+0000","last_fullsized":"2023-10-05T19:34:39.744802+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:01:07.958349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.745272+0000","last_change":"2023-10-05T19:34:39.745272+0000","last_active":"2023-10-05T19:34:39.745272+0000","last_peered":"2023-10-05T19:34:39.745272+0000","last_clean":"2023-10-05T19:34:39.745272+0000","last_became_active":"2023-10-05T19:34:39.744432+0000","last_became_peered":"2023-10-05T19:34:39.744432+0000","last_unstale":"2023-10-05T19:34:39.745272+0000","last_undegraded":"2023-10-05T19:34:39.745272+0000","last_fullsized":"2023-10-05T19:34:39.745272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:25.236881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742096+0000","last_change":"2023-10-05T19:34:39.742096+0000","last_active":"2023-10-05T19:34:39.742096+0000","last_peered":"2023-10-05T19:34:39.742096+0000","last_clean":"2023-10-05T19:34:39.742096+0000","last_became_active":"2023-10-05T19:34:39.741724+0000","last_became_peered":"2023-10-05T19:34:39.741724+0000","last_unstale":"2023-10-05T19:34:39.742096+0000","last_undegraded":"2023-10-05T19:34:39.742096+0000","last_fullsized":"2023-10-05T19:34:39.742096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:41:32.950188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744877+0000","last_change":"2023-10-05T19:34:39.744877+0000","last_active":"2023-10-05T19:34:39.744877+0000","last_peered":"2023-10-05T19:34:39.744877+0000","last_clean":"2023-10-05T19:34:39.744877+0000","last_became_active":"2023-10-05T19:34:39.744365+0000","last_became_peered":"2023-10-05T19:34:39.744365+0000","last_unstale":"2023-10-05T19:34:39.744877+0000","last_undegraded":"2023-10-05T19:34:39.744877+0000","last_fullsized":"2023-10-05T19:34:39.744877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:23:37.496987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744225+0000","last_change":"2023-10-05T19:34:39.742317+0000","last_active":"2023-10-05T19:34:39.744225+0000","last_peered":"2023-10-05T19:34:39.744225+0000","last_clean":"2023-10-05T19:34:39.744225+0000","last_became_active":"2023-10-05T19:34:39.741912+0000","last_became_peered":"2023-10-05T19:34:39.741912+0000","last_unstale":"2023-10-05T19:34:39.744225+0000","last_undegraded":"2023-10-05T19:34:39.744225+0000","last_fullsized":"2023-10-05T19:34:39.744225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:36:13.714700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742091+0000","last_change":"2023-10-05T19:34:39.742091+0000","last_active":"2023-10-05T19:34:39.742091+0000","last_peered":"2023-10-05T19:34:39.742091+0000","last_clean":"2023-10-05T19:34:39.742091+0000","last_became_active":"2023-10-05T19:34:39.741619+0000","last_became_peered":"2023-10-05T19:34:39.741619+0000","last_unstale":"2023-10-05T19:34:39.742091+0000","last_undegraded":"2023-10-05T19:34:39.742091+0000","last_fullsized":"2023-10-05T19:34:39.742091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:44:39.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742975+0000","last_change":"2023-10-05T19:34:39.742975+0000","last_active":"2023-10-05T19:34:39.742975+0000","last_peered":"2023-10-05T19:34:39.742975+0000","last_clean":"2023-10-05T19:34:39.742975+0000","last_became_active":"2023-10-05T19:34:39.742489+0000","last_became_peered":"2023-10-05T19:34:39.742489+0000","last_unstale":"2023-10-05T19:34:39.742975+0000","last_undegraded":"2023-10-05T19:34:39.742975+0000","last_fullsized":"2023-10-05T19:34:39.742975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:35:38.574309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.740394+0000","last_change":"2023-10-05T19:34:39.740394+0000","last_active":"2023-10-05T19:34:39.740394+0000","last_peered":"2023-10-05T19:34:39.740394+0000","last_clean":"2023-10-05T19:34:39.740394+0000","last_became_active":"2023-10-05T19:34:39.740174+0000","last_became_peered":"2023-10-05T19:34:39.740174+0000","last_unstale":"2023-10-05T19:34:39.740394+0000","last_undegraded":"2023-10-05T19:34:39.740394+0000","last_fullsized":"2023-10-05T19:34:39.740394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:01:41.744098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-05T19:34:38.720932+0000","last_change":"2023-10-05T19:34:33.210097+0000","last_active":"2023-10-05T19:34:38.720932+0000","last_peered":"2023-10-05T19:34:38.720932+0000","last_clean":"2023-10-05T19:34:38.720932+0000","last_became_active":"2023-10-05T19:34:33.208030+0000","last_became_peered":"2023-10-05T19:34:33.208030+0000","last_unstale":"2023-10-05T19:34:38.720932+0000","last_undegraded":"2023-10-05T19:34:38.720932+0000","last_fullsized":"2023-10-05T19:34:38.720932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:32.010201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:32.010201+0000","last_clean_scrub_stamp":"2023-10-05T19:34:32.010201+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T19:51:43.586421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559612,"statfs":{"total":107374182400,"available":107069042688,"internally_reserved":0,"allocated":716800,"data_stored":623851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559612,"statfs":{"total":107374182400,"available":107069042688,"internally_reserved":0,"allocated":716800,"data_stored":623851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560192,"statfs":{"total":107374182400,"available":107069636608,"internally_reserved":0,"allocated":122880,"data_stored":33483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:34:42.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:34:43.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:43.048 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-10-05T19:34:43.058 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-05T19:34:41.135644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893384,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679416,"statfs":{"total":322122547200,"available":321207721984,"internally_reserved":0,"allocated":1556480,"data_stored":1281185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001014"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744802+0000","last_change":"2023-10-05T19:34:39.744802+0000","last_active":"2023-10-05T19:34:39.744802+0000","last_peered":"2023-10-05T19:34:39.744802+0000","last_clean":"2023-10-05T19:34:39.744802+0000","last_became_active":"2023-10-05T19:34:39.744411+0000","last_became_peered":"2023-10-05T19:34:39.744411+0000","last_unstale":"2023-10-05T19:34:39.744802+0000","last_undegraded":"2023-10-05T19:34:39.744802+0000","last_fullsized":"2023-10-05T19:34:39.744802+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:01:07.958349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.745272+0000","last_change":"2023-10-05T19:34:39.745272+0000","last_active":"2023-10-05T19:34:39.745272+0000","last_peered":"2023-10-05T19:34:39.745272+0000","last_clean":"2023-10-05T19:34:39.745272+0000","last_became_active":"2023-10-05T19:34:39.744432+0000","last_became_peered":"2023-10-05T19:34:39.744432+0000","last_unstale":"2023-10-05T19:34:39.745272+0000","last_undegraded":"2023-10-05T19:34:39.745272+0000","last_fullsized":"2023-10-05T19:34:39.745272+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:25.236881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742096+0000","last_change":"2023-10-05T19:34:39.742096+0000","last_active":"2023-10-05T19:34:39.742096+0000","last_peered":"2023-10-05T19:34:39.742096+0000","last_clean":"2023-10-05T19:34:39.742096+0000","last_became_active":"2023-10-05T19:34:39.741724+0000","last_became_peered":"2023-10-05T19:34:39.741724+0000","last_unstale":"2023-10-05T19:34:39.742096+0000","last_undegraded":"2023-10-05T19:34:39.742096+0000","last_fullsized":"2023-10-05T19:34:39.742096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:41:32.950188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744877+0000","last_change":"2023-10-05T19:34:39.744877+0000","last_active":"2023-10-05T19:34:39.744877+0000","last_peered":"2023-10-05T19:34:39.744877+0000","last_clean":"2023-10-05T19:34:39.744877+0000","last_became_active":"2023-10-05T19:34:39.744365+0000","last_became_peered":"2023-10-05T19:34:39.744365+0000","last_unstale":"2023-10-05T19:34:39.744877+0000","last_undegraded":"2023-10-05T19:34:39.744877+0000","last_fullsized":"2023-10-05T19:34:39.744877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:23:37.496987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.744225+0000","last_change":"2023-10-05T19:34:39.742317+0000","last_active":"2023-10-05T19:34:39.744225+0000","last_peered":"2023-10-05T19:34:39.744225+0000","last_clean":"2023-10-05T19:34:39.744225+0000","last_became_active":"2023-10-05T19:34:39.741912+0000","last_became_peered":"2023-10-05T19:34:39.741912+0000","last_unstale":"2023-10-05T19:34:39.744225+0000","last_undegraded":"2023-10-05T19:34:39.744225+0000","last_fullsized":"2023-10-05T19:34:39.744225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:36:13.714700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742091+0000","last_change":"2023-10-05T19:34:39.742091+0000","last_active":"2023-10-05T19:34:39.742091+0000","last_peered":"2023-10-05T19:34:39.742091+0000","last_clean":"2023-10-05T19:34:39.742091+0000","last_became_active":"2023-10-05T19:34:39.741619+0000","last_became_peered":"2023-10-05T19:34:39.741619+0000","last_unstale":"2023-10-05T19:34:39.742091+0000","last_undegraded":"2023-10-05T19:34:39.742091+0000","last_fullsized":"2023-10-05T19:34:39.742091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:44:39.751094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.742975+0000","last_change":"2023-10-05T19:34:39.742975+0000","last_active":"2023-10-05T19:34:39.742975+0000","last_peered":"2023-10-05T19:34:39.742975+0000","last_clean":"2023-10-05T19:34:39.742975+0000","last_became_active":"2023-10-05T19:34:39.742489+0000","last_became_peered":"2023-10-05T19:34:39.742489+0000","last_unstale":"2023-10-05T19:34:39.742975+0000","last_undegraded":"2023-10-05T19:34:39.742975+0000","last_fullsized":"2023-10-05T19:34:39.742975+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:35:38.574309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:34:39.740394+0000","last_change":"2023-10-05T19:34:39.740394+0000","last_active":"2023-10-05T19:34:39.740394+0000","last_peered":"2023-10-05T19:34:39.740394+0000","last_clean":"2023-10-05T19:34:39.740394+0000","last_became_active":"2023-10-05T19:34:39.740174+0000","last_became_peered":"2023-10-05T19:34:39.740174+0000","last_unstale":"2023-10-05T19:34:39.740394+0000","last_undegraded":"2023-10-05T19:34:39.740394+0000","last_fullsized":"2023-10-05T19:34:39.740394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:38.713422+0000","last_clean_scrub_stamp":"2023-10-05T19:34:38.713422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:01:41.744098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-05T19:34:38.720932+0000","last_change":"2023-10-05T19:34:33.210097+0000","last_active":"2023-10-05T19:34:38.720932+0000","last_peered":"2023-10-05T19:34:38.720932+0000","last_clean":"2023-10-05T19:34:38.720932+0000","last_became_active":"2023-10-05T19:34:33.208030+0000","last_became_peered":"2023-10-05T19:34:33.208030+0000","last_unstale":"2023-10-05T19:34:38.720932+0000","last_undegraded":"2023-10-05T19:34:38.720932+0000","last_fullsized":"2023-10-05T19:34:38.720932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:34:32.010201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:34:32.010201+0000","last_clean_scrub_stamp":"2023-10-05T19:34:32.010201+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T19:51:43.586421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559612,"statfs":{"total":107374182400,"available":107069042688,"internally_reserved":0,"allocated":716800,"data_stored":623851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559612,"statfs":{"total":107374182400,"available":107069042688,"internally_reserved":0,"allocated":716800,"data_stored":623851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560192,"statfs":{"total":107374182400,"available":107069636608,"internally_reserved":0,"allocated":122880,"data_stored":33483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:34:43.059 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T19:34:43.059 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T19:34:43.059 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T19:34:43.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T19:34:43.380 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:34:43.380 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T19:34:43.393 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T19:34:43.394 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-10-05T19:34:43.406 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p /home/ubuntu/cephtest/ca 2023-10-05T19:34:43.414 DEBUG:teuthology.orchestra.run.smithi064:> openssl req -x509 -nodes -newkey rsa:4096 -keyout /home/ubuntu/cephtest/ca/kmiproot.key -days 365 -out /home/ubuntu/cephtest/ca/kmiproot.crt -subj /CN=kmiproot 2023-10-05T19:34:43.483 INFO:teuthology.orchestra.run.smithi064.stderr:Generating a RSA private key 2023-10-05T19:34:43.946 INFO:teuthology.orchestra.run.smithi064.stderr:....................................................................................................++++ 2023-10-05T19:34:44.610 INFO:teuthology.orchestra.run.smithi064.stderr:.....................................................................................................................................................++++ 2023-10-05T19:34:44.611 INFO:teuthology.orchestra.run.smithi064.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmiproot.key' 2023-10-05T19:34:44.611 INFO:teuthology.orchestra.run.smithi064.stderr:----- 2023-10-05T19:34:44.619 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p /home/ubuntu/cephtest/ca 2023-10-05T19:34:44.626 DEBUG:teuthology.orchestra.run.smithi064:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-client.key -out /home/ubuntu/cephtest/ca/kmip-client.csr -subj /CN=rgw-client 2023-10-05T19:34:44.636 INFO:teuthology.orchestra.run.smithi064.stderr:Generating a RSA private key 2023-10-05T19:34:44.660 INFO:teuthology.orchestra.run.smithi064.stderr:...................+++++ 2023-10-05T19:34:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:.....................+++++ 2023-10-05T19:34:44.682 INFO:teuthology.orchestra.run.smithi064.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-client.key' 2023-10-05T19:34:44.683 INFO:teuthology.orchestra.run.smithi064.stderr:----- 2023-10-05T19:34:44.684 DEBUG:teuthology.orchestra.run.smithi064:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-client.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-client.crt -days 365 -sha256 2023-10-05T19:34:44.691 INFO:teuthology.orchestra.run.smithi064.stderr:Signature ok 2023-10-05T19:34:44.691 INFO:teuthology.orchestra.run.smithi064.stderr:subject=CN = rgw-client 2023-10-05T19:34:44.691 INFO:teuthology.orchestra.run.smithi064.stderr:Getting CA Private Key 2023-10-05T19:34:44.701 DEBUG:teuthology.orchestra.run.smithi064:> rm /home/ubuntu/cephtest/ca/kmip-client.csr /home/ubuntu/cephtest/ca/kmiproot.srl 2023-10-05T19:34:44.706 DEBUG:teuthology.orchestra.run.smithi064:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-client.crt 2023-10-05T19:34:44.759 INFO:teuthology.orchestra.run.smithi064.stdout:/home/ubuntu/cephtest/ca/kmip-client.crt: OK 2023-10-05T19:34:44.760 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p /home/ubuntu/cephtest/ca 2023-10-05T19:34:44.767 DEBUG:teuthology.orchestra.run.smithi064:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-server.key -out /home/ubuntu/cephtest/ca/kmip-server.csr -subj /CN=smithi064.front.sepia.ceph.com 2023-10-05T19:34:44.775 INFO:teuthology.orchestra.run.smithi064.stderr:Generating a RSA private key 2023-10-05T19:34:44.800 INFO:teuthology.orchestra.run.smithi064.stderr:....................+++++ 2023-10-05T19:34:44.810 INFO:teuthology.orchestra.run.smithi064.stderr:.........+++++ 2023-10-05T19:34:44.810 INFO:teuthology.orchestra.run.smithi064.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-server.key' 2023-10-05T19:34:44.810 INFO:teuthology.orchestra.run.smithi064.stderr:----- 2023-10-05T19:34:44.812 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:34:44.812 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ca/kmip-server.ext 2023-10-05T19:34:44.817 DEBUG:teuthology.orchestra.run.smithi064:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-server.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-server.crt -days 365 -sha256 -extfile /home/ubuntu/cephtest/ca/kmip-server.ext 2023-10-05T19:34:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:Signature ok 2023-10-05T19:34:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:subject=CN = smithi064.front.sepia.ceph.com 2023-10-05T19:34:44.867 INFO:teuthology.orchestra.run.smithi064.stderr:Getting CA Private Key 2023-10-05T19:34:44.879 DEBUG:teuthology.orchestra.run.smithi064:> rm /home/ubuntu/cephtest/ca/kmip-server.csr /home/ubuntu/cephtest/ca/kmiproot.srl /home/ubuntu/cephtest/ca/kmip-server.ext 2023-10-05T19:34:44.887 DEBUG:teuthology.orchestra.run.smithi064:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-server.crt 2023-10-05T19:34:44.896 INFO:teuthology.orchestra.run.smithi064.stdout:/home/ubuntu/cephtest/ca/kmip-server.crt: OK 2023-10-05T19:34:44.897 INFO:teuthology.run_tasks:Running task exec... 2023-10-05T19:34:44.910 INFO:teuthology.task.exec:Executing custom commands... 2023-10-05T19:34:44.910 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:34:44.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key' 2023-10-05T19:34:44.925 INFO:teuthology.run_tasks:Running task pykmip... 2023-10-05T19:34:44.940 DEBUG:tasks.pykmip:PyKMIP config is {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}} 2023-10-05T19:34:44.940 INFO:tasks.pykmip:Downloading pykmip... 2023-10-05T19:34:44.941 INFO:tasks.pykmip:Using branch 'master' for pykmip 2023-10-05T19:34:44.941 INFO:tasks.pykmip:sha1=None 2023-10-05T19:34:44.941 DEBUG:teuthology.orchestra.run.smithi064:> git clone -b master https://github.com/OpenKMIP/PyKMIP /home/ubuntu/cephtest/pykmip 2023-10-05T19:34:44.948 INFO:teuthology.orchestra.run.smithi064.stderr:Cloning into '/home/ubuntu/cephtest/pykmip'... 2023-10-05T19:34:45.583 INFO:tasks.pykmip:Setting up virtualenv for pykmip... 2023-10-05T19:34:45.583 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && python3 -m venv .pykmipenv 2023-10-05T19:34:47.338 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install --upgrade pip 2023-10-05T19:34:48.549 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pip 2023-10-05T19:34:48.629 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading pip-23.2.1-py3-none-any.whl (2.1 MB) 2023-10-05T19:34:48.875 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: pip 2023-10-05T19:34:48.875 INFO:teuthology.orchestra.run.smithi064.stdout: Attempting uninstall: pip 2023-10-05T19:34:48.875 INFO:teuthology.orchestra.run.smithi064.stdout: Found existing installation: pip 20.0.2 2023-10-05T19:34:48.934 INFO:teuthology.orchestra.run.smithi064.stdout: Uninstalling pip-20.0.2: 2023-10-05T19:34:48.935 INFO:teuthology.orchestra.run.smithi064.stdout: Successfully uninstalled pip-20.0.2 2023-10-05T19:34:49.875 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed pip-23.2.1 2023-10-05T19:34:49.919 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install pytz -e /home/ubuntu/cephtest/pykmip 2023-10-05T19:34:50.324 INFO:teuthology.orchestra.run.smithi064.stdout:Obtaining file:///home/ubuntu/cephtest/pykmip 2023-10-05T19:34:50.328 INFO:teuthology.orchestra.run.smithi064.stdout: Installing build dependencies: started 2023-10-05T19:34:52.447 INFO:teuthology.orchestra.run.smithi064.stdout: Installing build dependencies: finished with status 'done' 2023-10-05T19:34:52.449 INFO:teuthology.orchestra.run.smithi064.stdout: Checking if build backend supports build_editable: started 2023-10-05T19:34:52.571 INFO:teuthology.orchestra.run.smithi064.stdout: Checking if build backend supports build_editable: finished with status 'done' 2023-10-05T19:34:52.573 INFO:teuthology.orchestra.run.smithi064.stdout: Getting requirements to build editable: started 2023-10-05T19:34:52.761 INFO:teuthology.orchestra.run.smithi064.stdout: Getting requirements to build editable: finished with status 'done' 2023-10-05T19:34:52.766 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing editable metadata (pyproject.toml): started 2023-10-05T19:34:52.971 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing editable metadata (pyproject.toml): finished with status 'done' 2023-10-05T19:34:53.169 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pytz 2023-10-05T19:34:53.169 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for pytz from https://files.pythonhosted.org/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl.metadata 2023-10-05T19:34:53.234 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB) 2023-10-05T19:34:53.615 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting cryptography (from PyKMIP==0.11.0.dev1) 2023-10-05T19:34:53.615 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for cryptography from https://files.pythonhosted.org/packages/eb/4b/f86cc66c632cf0948ca1712aadd255f624deef1cd371ea3bfd30851e188d/cryptography-41.0.4-cp37-abi3-manylinux_2_28_x86_64.whl.metadata 2023-10-05T19:34:53.633 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading cryptography-41.0.4-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (5.2 kB) 2023-10-05T19:34:53.715 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting enum-compat (from PyKMIP==0.11.0.dev1) 2023-10-05T19:34:53.734 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading enum_compat-0.0.3-py3-none-any.whl (1.3 kB) 2023-10-05T19:34:53.868 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting requests (from PyKMIP==0.11.0.dev1) 2023-10-05T19:34:53.868 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for requests from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata 2023-10-05T19:34:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) 2023-10-05T19:34:53.974 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting six (from PyKMIP==0.11.0.dev1) 2023-10-05T19:34:53.992 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T19:34:54.596 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting sqlalchemy (from PyKMIP==0.11.0.dev1) 2023-10-05T19:34:54.596 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for sqlalchemy from https://files.pythonhosted.org/packages/7c/70/5a36e9dd9bb62dca4d07d811c993eadd432769f541ea91c851f41f9d5ee8/SQLAlchemy-2.0.21-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:34:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading SQLAlchemy-2.0.21-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.4 kB) 2023-10-05T19:34:54.945 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting cffi>=1.12 (from cryptography->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:54.945 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for cffi>=1.12 from https://files.pythonhosted.org/packages/f1/c9/326611aa83e16b13b6db4dbb73b5455c668159a003c4c2f0c3bcb2ddabaf/cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:34:54.963 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB) 2023-10-05T19:34:55.162 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting charset-normalizer<4,>=2 (from requests->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:55.162 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/1e/c8/fd52271326c052f95f47ef718b018aa2bc3fd097d9bac44d7d48894c6130/charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:34:55.180 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (32 kB) 2023-10-05T19:34:55.330 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting idna<4,>=2.5 (from requests->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:55.348 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T19:34:55.386 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 61.5/61.5 kB 1.5 MB/s eta 0:00:00 2023-10-05T19:34:55.531 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting urllib3<3,>=1.21.1 (from requests->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:55.531 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/26/40/9957270221b6d3e9a3b92fdfba80dd5c9661ff45a664b47edd5d00f707f5/urllib3-2.0.6-py3-none-any.whl.metadata 2023-10-05T19:34:55.548 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading urllib3-2.0.6-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:34:55.638 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting certifi>=2017.4.17 (from requests->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:55.638 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata 2023-10-05T19:34:55.664 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB) 2023-10-05T19:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting typing-extensions>=4.2.0 (from sqlalchemy->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:55.791 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for typing-extensions>=4.2.0 from https://files.pythonhosted.org/packages/24/21/7d397a4b7934ff4028987914ac1044d3b7d52712f30e2ac7a2ae5bc86dd0/typing_extensions-4.8.0-py3-none-any.whl.metadata 2023-10-05T19:34:55.809 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB) 2023-10-05T19:34:56.143 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting greenlet!=0.4.17 (from sqlalchemy->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for greenlet!=0.4.17 from https://files.pythonhosted.org/packages/75/5a/4ecc17749c54882dd3422befcd35fec5128ae2a72138844899322af0b55a/greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata 2023-10-05T19:34:56.160 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.8 kB) 2023-10-05T19:34:56.260 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pycparser (from cffi>=1.12->cryptography->PyKMIP==0.11.0.dev1) 2023-10-05T19:34:56.278 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) 2023-10-05T19:34:56.317 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 118.7/118.7 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:34:56.376 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T19:34:56.451 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 502.5/502.5 kB 7.1 MB/s eta 0:00:00 2023-10-05T19:34:56.473 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading cryptography-41.0.4-cp37-abi3-manylinux_2_28_x86_64.whl (4.4 MB) 2023-10-05T19:34:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 4.4/4.4 MB 15.6 MB/s eta 0:00:00 2023-10-05T19:34:56.780 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading requests-2.31.0-py3-none-any.whl (62 kB) 2023-10-05T19:34:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 62.6/62.6 kB 2.2 MB/s eta 0:00:00 2023-10-05T19:34:56.832 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading SQLAlchemy-2.0.21-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB) 2023-10-05T19:34:57.036 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 3.0/3.0 MB 15.3 MB/s eta 0:00:00 2023-10-05T19:34:57.073 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T19:34:57.119 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 158.3/158.3 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:34:57.142 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (444 kB) 2023-10-05T19:34:57.203 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 444.7/444.7 kB 7.8 MB/s eta 0:00:00 2023-10-05T19:34:57.224 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (137 kB) 2023-10-05T19:34:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 137.9/137.9 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:34:57.306 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (617 kB) 2023-10-05T19:34:57.370 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 617.1/617.1 kB 10.2 MB/s eta 0:00:00 2023-10-05T19:34:57.391 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB) 2023-10-05T19:34:57.479 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading urllib3-2.0.6-py3-none-any.whl (123 kB) 2023-10-05T19:34:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 123.8/123.8 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:34:57.559 INFO:teuthology.orchestra.run.smithi064.stdout:Building wheels for collected packages: PyKMIP 2023-10-05T19:34:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: Building editable for PyKMIP (pyproject.toml): started 2023-10-05T19:34:57.791 INFO:teuthology.orchestra.run.smithi064.stdout: Building editable for PyKMIP (pyproject.toml): finished with status 'done' 2023-10-05T19:34:57.793 INFO:teuthology.orchestra.run.smithi064.stdout: Created wheel for PyKMIP: filename=PyKMIP-0.11.0.dev1-0.editable-py3-none-any.whl size=7835 sha256=dda401c9f0eca6ef4ce3a0d9e800cf6f0b2137f45cce5d49f583ca55285d1921 2023-10-05T19:34:57.793 INFO:teuthology.orchestra.run.smithi064.stdout: Stored in directory: /tmp/pip-ephem-wheel-cache-iapfftlc/wheels/ea/d8/28/3faf5b092e89e900bdd64c179a1cc2d58f85008fc572ca64af 2023-10-05T19:34:57.798 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully built PyKMIP 2023-10-05T19:34:57.881 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: pytz, enum-compat, urllib3, typing-extensions, six, pycparser, idna, greenlet, charset-normalizer, certifi, sqlalchemy, requests, cffi, cryptography, PyKMIP 2023-10-05T19:35:00.590 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed PyKMIP-0.11.0.dev1 certifi-2023.7.22 cffi-1.16.0 charset-normalizer-3.3.0 cryptography-41.0.4 enum-compat-0.0.3 greenlet-3.0.0 idna-3.4 pycparser-2.21 pytz-2023.3.post1 requests-2.31.0 six-1.16.0 sqlalchemy-2.0.21 typing-extensions-4.8.0 urllib3-2.0.6 2023-10-05T19:35:00.756 INFO:tasks.pykmip:Installing system dependenies for PyKMIP... 2023-10-05T19:35:00.781 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install bindep 2023-10-05T19:35:01.283 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting bindep 2023-10-05T19:35:01.346 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading bindep-2.11.0-py2.py3-none-any.whl (34 kB) 2023-10-05T19:35:01.496 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting Parsley (from bindep) 2023-10-05T19:35:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading Parsley-1.3-py2.py3-none-any.whl (88 kB) 2023-10-05T19:35:01.564 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 88.9/88.9 kB 1.9 MB/s eta 0:00:00 2023-10-05T19:35:01.658 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pbr>=2.0.0 (from bindep) 2023-10-05T19:35:01.675 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading pbr-5.11.1-py2.py3-none-any.whl (112 kB) 2023-10-05T19:35:01.735 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 112.7/112.7 kB 1.8 MB/s eta 0:00:00 2023-10-05T19:35:01.806 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting distro>=1.7.0 (from bindep) 2023-10-05T19:35:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading distro-1.8.0-py3-none-any.whl (20 kB) 2023-10-05T19:35:01.931 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting packaging (from bindep) 2023-10-05T19:35:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl.metadata 2023-10-05T19:35:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB) 2023-10-05T19:35:02.010 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading packaging-23.2-py3-none-any.whl (53 kB) 2023-10-05T19:35:02.067 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 53.0/53.0 kB 819.0 kB/s eta 0:00:00 2023-10-05T19:35:02.161 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: Parsley, pbr, packaging, distro, bindep 2023-10-05T19:35:02.680 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed Parsley-1.3 bindep-2.11.0 distro-1.8.0 packaging-23.2 pbr-5.11.1 2023-10-05T19:35:02.759 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && bindep --brief --file /tmp/bindep-vohh4mw5.txt 2023-10-05T19:35:03.359 INFO:teuthology.orchestra.run.smithi064.stdout:libffi-dev 2023-10-05T19:35:03.359 INFO:teuthology.orchestra.run.smithi064.stdout:libsqlite3-dev 2023-10-05T19:35:03.359 INFO:teuthology.orchestra.run.smithi064.stdout:libssl-dev 2023-10-05T19:35:03.359 INFO:teuthology.orchestra.run.smithi064.stdout:python-dev 2023-10-05T19:35:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:35:03.532 INFO:teuthology.packaging:Installing package libffi-dev on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:35:03.532 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libffi-dev 2023-10-05T19:35:03.607 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:35:03.815 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:35:03.816 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:35:04.007 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:35:04.008 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:35:04.008 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:35:04.039 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:35:04.040 INFO:teuthology.orchestra.run.smithi064.stdout: libffi-dev 2023-10-05T19:35:04.227 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 1 newly installed, 0 to remove and 253 not upgraded. 2023-10-05T19:35:04.227 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 57.0 kB of archives. 2023-10-05T19:35:04.227 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 312 kB of additional disk space will be used. 2023-10-05T19:35:04.227 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libffi-dev amd64 3.3-4 [57.0 kB] 2023-10-05T19:35:04.781 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 57.0 kB in 0s (156 kB/s) 2023-10-05T19:35:04.880 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libffi-dev:amd64. 2023-10-05T19:35:04.940 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152592 files and directories currently installed.) 2023-10-05T19:35:04.945 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../libffi-dev_3.3-4_amd64.deb ... 2023-10-05T19:35:04.995 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libffi-dev:amd64 (3.3-4) ... 2023-10-05T19:35:05.449 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libffi-dev:amd64 (3.3-4) ... 2023-10-05T19:35:05.610 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-10-05T19:35:06.170 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:35:07.747 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:35:07.750 INFO:teuthology.packaging:Installing package libsqlite3-dev on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:35:07.750 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libsqlite3-dev 2023-10-05T19:35:07.821 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:35:08.029 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:35:08.030 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:35:08.231 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:35:08.232 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:35:08.232 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:35:08.232 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-05T19:35:08.233 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-0 2023-10-05T19:35:08.234 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-05T19:35:08.235 INFO:teuthology.orchestra.run.smithi064.stdout: sqlite3-doc 2023-10-05T19:35:08.268 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:35:08.269 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-dev 2023-10-05T19:35:08.270 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-05T19:35:08.271 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-0 2023-10-05T19:35:08.455 INFO:teuthology.orchestra.run.smithi064.stdout:1 upgraded, 1 newly installed, 0 to remove and 252 not upgraded. 2023-10-05T19:35:08.455 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 1,245 kB of archives. 2023-10-05T19:35:08.455 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 2,372 kB of additional disk space will be used. 2023-10-05T19:35:08.455 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.5 [549 kB] 2023-10-05T19:35:09.005 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-dev amd64 3.31.1-4ubuntu0.5 [697 kB] 2023-10-05T19:35:09.476 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 1,245 kB in 1s (1,516 kB/s) 2023-10-05T19:35:09.757 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152626 files and directories currently installed.) 2023-10-05T19:35:09.762 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../libsqlite3-0_3.31.1-4ubuntu0.5_amd64.deb ... 2023-10-05T19:35:09.966 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) over (3.31.1-4ubuntu0.3) ... 2023-10-05T19:35:10.370 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-dev:amd64. 2023-10-05T19:35:10.390 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../libsqlite3-dev_3.31.1-4ubuntu0.5_amd64.deb ... 2023-10-05T19:35:10.428 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-10-05T19:35:10.820 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) ... 2023-10-05T19:35:10.941 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-10-05T19:35:11.092 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:35:11.570 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:35:11.572 INFO:teuthology.packaging:Installing package libssl-dev on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:35:11.572 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libssl-dev 2023-10-05T19:35:11.643 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:35:11.854 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:35:11.854 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:35:12.050 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:35:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:35:12.050 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:35:12.051 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-05T19:35:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: libssl1.1 2023-10-05T19:35:12.053 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-05T19:35:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: libssl-doc 2023-10-05T19:35:12.088 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:35:12.089 INFO:teuthology.orchestra.run.smithi064.stdout: libssl-dev 2023-10-05T19:35:12.090 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-05T19:35:12.091 INFO:teuthology.orchestra.run.smithi064.stdout: libssl1.1 2023-10-05T19:35:12.273 INFO:teuthology.orchestra.run.smithi064.stdout:1 upgraded, 1 newly installed, 0 to remove and 251 not upgraded. 2023-10-05T19:35:12.273 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 2,905 kB of archives. 2023-10-05T19:35:12.273 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 8,011 kB of additional disk space will be used. 2023-10-05T19:35:12.273 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.19 [1,321 kB] 2023-10-05T19:35:12.926 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.19 [1,584 kB] 2023-10-05T19:35:13.345 INFO:teuthology.orchestra.run.smithi064.stdout:Preconfiguring packages ... 2023-10-05T19:35:13.493 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 2,905 kB in 1s (3,160 kB/s) 2023-10-05T19:35:13.802 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152635 files and directories currently installed.) 2023-10-05T19:35:13.807 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-05T19:35:14.031 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.19) over (1.1.1f-1ubuntu2.16) ... 2023-10-05T19:35:14.522 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-05T19:35:14.541 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../libssl-dev_1.1.1f-1ubuntu2.19_amd64.deb ... 2023-10-05T19:35:14.593 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-05T19:35:15.086 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-05T19:35:15.473 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-05T19:35:15.600 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:35:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:35:16.060 INFO:teuthology.packaging:Installing package python-dev on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:35:16.060 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev 2023-10-05T19:35:16.136 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:35:16.347 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:35:16.469 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:35:16.540 INFO:teuthology.orchestra.run.smithi064.stdout:python-dev-is-python2 is already the newest version (2.7.17-4). 2023-10-05T19:35:16.630 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:35:16.630 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:35:16.630 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:35:16.631 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 251 not upgraded. 2023-10-05T19:35:16.631 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:35:16.632 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/pykmip && cp /home/ubuntu/cephtest/pykmip/examples/policy.json /home/ubuntu/cephtest/pykmip 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#0 cclient='client.0' cconfig={'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'} 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#1 ip,p,h 172.21.15.64 5696 smithi064.front.sepia.ceph.com 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#2 clientca kmiproot 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#3 servercert kmip-server 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#4 servercert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-server.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-server.key', key_type='rsa:2048', name='kmip-server', remote=Remote(name='ubuntu@smithi064.front.sepia.ceph.com')) 2023-10-05T19:35:16.639 INFO:tasks.pykmip:#3 clientcert kmip-client 2023-10-05T19:35:16.640 INFO:tasks.pykmip:#4 clientcert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-client.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-client.key', key_type='rsa:2048', name='kmip-client', remote=Remote(name='ubuntu@smithi064.front.sepia.ceph.com')) 2023-10-05T19:35:16.640 INFO:tasks.pykmip:#5 clientca Namespace(certificate='/home/ubuntu/cephtest/ca/kmiproot.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmiproot.key', key_type='rsa:4096', name='kmiproot', remote=Remote(name='ubuntu@smithi064.front.sepia.ceph.com')) 2023-10-05T19:35:16.640 INFO:tasks.pykmip:#6 serverkey /home/ubuntu/cephtest/ca/kmip-server.key servercert /home/ubuntu/cephtest/ca/kmip-server.crt 2023-10-05T19:35:16.640 INFO:tasks.pykmip:#6 clientkey /home/ubuntu/cephtest/ca/kmip-client.key clientcert /home/ubuntu/cephtest/ca/kmip-client.crt 2023-10-05T19:35:16.640 INFO:tasks.pykmip:#7 clientca /home/ubuntu/cephtest/ca/kmiproot.crt 2023-10-05T19:35:16.693 INFO:tasks.pykmip:Running pykmip... 2023-10-05T19:35:16.693 INFO:tasks.pykmip.client.0:Restarting daemon 2023-10-05T19:35:16.694 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/pykmip && exec bash -c 'cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && HOME=/home/ubuntu/cephtest/pykmip && exec pykmip-server -f pykmip.conf -l /home/ubuntu/cephtest/pykmip/pykmip.log & { read; kill %1; }') 2023-10-05T19:35:16.696 INFO:tasks.pykmip.client.0:Started 2023-10-05T19:35:26.697 DEBUG:teuthology.orchestra.run.smithi064:> . cephtest/pykmip/.pykmipenv/bin/activate;python 2023-10-05T19:35:27.450 INFO:teuthology.orchestra.run.smithi064.stdout:[{"Unique Identifier": "1", "Name": "pykmip-my-key-1", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Thu Oct 5 19:35:27 2023"}, {"Unique Identifier": "2", "Name": "pykmip-my-key-2", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Thu Oct 5 19:35:27 2023"}] 2023-10-05T19:35:27.521 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T19:35:27.536 DEBUG:tasks.rgw:config is {'client.0': {'use-pykmip-role': 'client.0'}} 2023-10-05T19:35:27.536 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T19:35:27.536 INFO:tasks.rgw:Creating data pools 2023-10-05T19:35:27.536 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T19:35:27.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-10-05T19:35:28.765 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T19:35:28.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T19:35:29.805 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T19:35:29.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T19:35:30.791 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T19:35:30.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T19:35:31.804 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T19:35:31.821 DEBUG:tasks.rgw:Pools created 2023-10-05T19:35:31.821 INFO:tasks.rgw:Starting rgw... 2023-10-05T19:35:31.821 INFO:tasks.rgw:rgw client.0 config is {'use-pykmip-role': 'client.0'} 2023-10-05T19:35:31.822 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T19:35:31.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.crt /etc/ceph/ 2023-10-05T19:35:31.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.key /etc/ceph/ 2023-10-05T19:35:31.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.crt /etc/ceph/ 2023-10-05T19:35:31.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.key /etc/ceph/ 2023-10-05T19:35:31.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.key /etc/ceph/ 2023-10-05T19:35:31.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.crt /etc/ceph/ 2023-10-05T19:35:31.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 600 /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-10-05T19:35:31.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown ceph /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-10-05T19:35:32.009 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T19:35:32.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.64:5696 --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T19:35:32.012 INFO:tasks.rgw.client.0:Started 2023-10-05T19:35:32.012 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:32.012 DEBUG:teuthology.orchestra.run.smithi064:> curl http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:32.049 INFO:teuthology.orchestra.run.smithi064.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:35:32.049 INFO:teuthology.orchestra.run.smithi064.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:35:32.050 INFO:teuthology.orchestra.run.smithi064.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:35:32.050 INFO:teuthology.orchestra.run.smithi064.stderr:curl: (7) Failed to connect to smithi064.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:35:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:35:33.052 DEBUG:teuthology.orchestra.run.smithi064:> curl http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:33.069 INFO:teuthology.orchestra.run.smithi064.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:35:33.069 INFO:teuthology.orchestra.run.smithi064.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:35:33.070 INFO:teuthology.orchestra.run.smithi064.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:35:33.070 INFO:teuthology.orchestra.run.smithi064.stderr:curl: (7) Failed to connect to smithi064.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:35:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:35:35.072 DEBUG:teuthology.orchestra.run.smithi064:> curl http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:35.088 INFO:teuthology.orchestra.run.smithi064.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:35:35.089 INFO:teuthology.orchestra.run.smithi064.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:35:35.089 INFO:teuthology.orchestra.run.smithi064.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:35:35.089 INFO:teuthology.orchestra.run.smithi064.stderr:curl: (7) Failed to connect to smithi064.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:35:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:35:39.091 DEBUG:teuthology.orchestra.run.smithi064:> curl http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:39.110 INFO:teuthology.orchestra.run.smithi064.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:35:39.110 INFO:teuthology.orchestra.run.smithi064.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:35:39.110 INFO:teuthology.orchestra.run.smithi064.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:35:39.110 INFO:teuthology.orchestra.run.smithi064.stderr:curl: (7) Failed to connect to smithi064.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:35:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:35:47.112 DEBUG:teuthology.orchestra.run.smithi064:> curl http://smithi064.front.sepia.ceph.com:80/ 2023-10-05T19:35:47.129 INFO:teuthology.orchestra.run.smithi064.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:35:47.129 INFO:teuthology.orchestra.run.smithi064.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:35:47.132 INFO:teuthology.orchestra.run.smithi064.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-10-05T19:35:47.134 INFO:teuthology.orchestra.run.smithi064.stdout:anonymous 2023-10-05T19:35:47.134 INFO:teuthology.run_tasks:Running task s3tests... 2023-10-05T19:35:47.148 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-10-05T19:35:47.149 INFO:tasks.s3tests:Downloading s3-tests... 2023-10-05T19:35:47.149 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-10-05T19:35:47.149 DEBUG:teuthology.orchestra.run.smithi064:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:35:47.156 INFO:teuthology.orchestra.run.smithi064.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2023-10-05T19:35:47.924 INFO:tasks.s3tests:Creating rgw users... 2023-10-05T19:35:47.924 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-10-05T19:35:47.925 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key BHALCIPGHFAQINSURGKR --secret vtUdnxzN9WNeLqe2/xKJt5x8MkT6tHnr7dFPn/Bsaspa5O3fmXD9nA== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:35:47.971 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:47.971 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:47.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.978+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:47.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:47.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 realm 2023-10-05T19:35:47.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:47.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:47.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:47.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:47.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:47.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:47.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:47.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.982+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:47.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.986+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:47.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:47.986+0000 7f5d52522e40 20 searching for the correct realm 2023-10-05T19:35:48.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:48.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:48.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:48.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:48.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:48.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.012 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.012 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.006+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:48.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:48.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:48.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:48.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:48.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.030+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 zone default found 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 4 Realm: () 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 zonegroup default 2023-10-05T19:35:48.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 started sync module instance, tier type = 2023-10-05T19:35:48.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.034+0000 7f5d52522e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:48.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=0 2023-10-05T19:35:48.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=1 2023-10-05T19:35:48.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=2 2023-10-05T19:35:48.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=3 2023-10-05T19:35:48.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=4 2023-10-05T19:35:48.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=5 2023-10-05T19:35:48.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=6 2023-10-05T19:35:48.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 20 add_watcher() i=7 2023-10-05T19:35:48.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.054+0000 7f5d52522e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:48.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.058+0000 7f5d20ff1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:48.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.058+0000 7f5d52522e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:48.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.058+0000 7f5d52522e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:48.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.062+0000 7f5d52522e40 5 note: GC not initialized 2023-10-05T19:35:48.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5c8f7fe700 20 reqs_thread_entry: start 2023-10-05T19:35:48.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:48.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5c8d7fa700 20 reqs_thread_entry: start 2023-10-05T19:35:48.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:48.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5c7affd700 20 reqs_thread_entry: start 2023-10-05T19:35:48.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 get_system_obj_state: rctx=0x7ffe2fc9fb38 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55b87eb480b0 s->prefetch_data=0 2023-10-05T19:35:48.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:35:48.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5c7b7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:18 2023 2023-10-05T19:35:48.071 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 get_system_obj_state: rctx=0x7ffe2fc9f828 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55b87eb480b0 s->prefetch_data=0 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 get_system_obj_state: rctx=0x7ffe2fc9f828 obj=default.rgw.meta:users.keys:BHALCIPGHFAQINSURGKR state=0x55b87eb480b0 s->prefetch_data=0 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache get: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR : miss 2023-10-05T19:35:48.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR info.flags=0x0 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 adding default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR to cache LRU end 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 get_system_obj_state: rctx=0x7ffe2fc9f0c8 obj=default.rgw.meta:users.keys:BHALCIPGHFAQINSURGKR state=0x55b87eb480b0 s->prefetch_data=0 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache get: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR : hit (negative entry) 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 20 get_system_obj_state: rctx=0x7ffe2fc9f768 obj=default.rgw.meta:users.keys:BHALCIPGHFAQINSURGKR state=0x55b87eb480b0 s->prefetch_data=0 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.066+0000 7f5d52522e40 10 cache get: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR : hit (negative entry) 2023-10-05T19:35:48.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:35:48.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d52522e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:48.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d52522e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:35:48.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d4e053700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94250889553568 notifier 4211 bl.length()=640 2023-10-05T19:35:48.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d4e053700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:35:48.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.070+0000 7f5d4e053700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:48.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:48.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:35:48.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T19:35:48.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d4d852700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94250889580608 notifier 4211 bl.length()=194 2023-10-05T19:35:48.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d4d852700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:48.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d4d852700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:35:48.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 cache put: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR info.flags=0x7 2023-10-05T19:35:48.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 moving default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR to cache LRU end 2023-10-05T19:35:48.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.074+0000 7f5d52522e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:BHALCIPGHFAQINSURGKR, ofs0, ns] 2023-10-05T19:35:48.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.078+0000 7f5d4e053700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94250889564320 notifier 4211 bl.length()=186 2023-10-05T19:35:48.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.078+0000 7f5d4e053700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR info.flags=0x7 2023-10-05T19:35:48.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.078+0000 7f5d4e053700 10 rgw watcher librados: moving default.rgw.meta+users.keys+BHALCIPGHFAQINSURGKR to cache LRU end 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "user_id": "foo.client.0", 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "display_name": "Mr. foo.client.0", 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "email": "foo.client.0+test@test.test", 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "suspended": 0, 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "max_buckets": 1000, 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "subusers": [], 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "keys": [ 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "user": "foo.client.0", 2023-10-05T19:35:48.082 INFO:teuthology.orchestra.run.smithi064.stdout: "access_key": "BHALCIPGHFAQINSURGKR", 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "secret_key": "vtUdnxzN9WNeLqe2/xKJt5x8MkT6tHnr7dFPn/Bsaspa5O3fmXD9nA==" 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "swift_keys": [], 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": [ 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "user-policy", 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "perm": "*" 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "op_mask": "read, write, delete", 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "default_placement": "", 2023-10-05T19:35:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: "default_storage_class": "", 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_tags": [], 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_quota": { 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "user_quota": { 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:48.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: "temp_url_keys": [], 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "rgw", 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: "mfa_ids": [] 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.078+0000 7f5d52522e40 20 remove_watcher() i=0 2023-10-05T19:35:48.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.078+0000 7f5d52522e40 2 removed watcher, disabling cache 2023-10-05T19:35:48.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.082+0000 7f5d52522e40 20 remove_watcher() i=1 2023-10-05T19:35:48.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.086+0000 7f5d52522e40 20 remove_watcher() i=2 2023-10-05T19:35:48.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.090+0000 7f5d52522e40 20 remove_watcher() i=3 2023-10-05T19:35:48.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.090+0000 7f5d52522e40 20 remove_watcher() i=4 2023-10-05T19:35:48.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.094+0000 7f5d52522e40 20 remove_watcher() i=5 2023-10-05T19:35:48.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.098+0000 7f5d52522e40 20 remove_watcher() i=6 2023-10-05T19:35:48.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.098+0000 7f5d52522e40 20 remove_watcher() i=7 2023-10-05T19:35:48.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.102+0000 7f5d20ff1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:48.115 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 6435655929 --totp-seed 4DTOWGZFDXJRVHNYOGGDAKXXM7OEHR73F5U347FWFW5TCNCAPP42YNSBOQ6GYODR --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:35:48.163 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:48.163 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:48.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 realm 2023-10-05T19:35:48.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.174+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.178+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.178+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.178+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:48.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.178+0000 7f5a74641e40 20 searching for the correct realm 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:48.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.202+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.209 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.206+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.209 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.206+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.209 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.206+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:48.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:48.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:48.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:48.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:48.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:48.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.234+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 zone default found 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 4 Realm: () 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:48.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 zonegroup default 2023-10-05T19:35:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:48.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 started sync module instance, tier type = 2023-10-05T19:35:48.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.238+0000 7f5a74641e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:48.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=0 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=1 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=2 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=3 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=4 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=5 2023-10-05T19:35:48.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=6 2023-10-05T19:35:48.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 20 add_watcher() i=7 2023-10-05T19:35:48.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.274+0000 7f5a74641e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:48.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.278+0000 7f5a74641e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:48.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.278+0000 7f5a74641e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:48.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.282+0000 7f5a43fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:48.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.290+0000 7f5a74641e40 5 note: GC not initialized 2023-10-05T19:35:48.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f59b2ffd700 20 reqs_thread_entry: start 2023-10-05T19:35:48.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f59b0ff9700 20 reqs_thread_entry: start 2023-10-05T19:35:48.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f59a6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:18 2023 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 get_system_obj_state: rctx=0x7ffc360e39c8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55cdaf39f0b0 s->prefetch_data=0 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:35:48.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f59a67fc700 20 reqs_thread_entry: start 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:35:48.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:48.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:35:48.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T19:35:48.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:48.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:48.294+0000 7f5a74641e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:49.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a74641e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:35:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a74641e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a74641e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:35:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a6b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018881 cookie 94341897924336 notifier 4217 bl.length()=654 2023-10-05T19:35:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a6b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:35:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a6b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a74641e40 20 remove_watcher() i=0 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.786+0000 7f5a74641e40 2 removed watcher, disabling cache 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.790+0000 7f5a74641e40 20 remove_watcher() i=1 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.790+0000 7f5a74641e40 20 remove_watcher() i=2 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.794+0000 7f5a74641e40 20 remove_watcher() i=3 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.794+0000 7f5a74641e40 20 remove_watcher() i=4 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.798+0000 7f5a74641e40 20 remove_watcher() i=5 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.802+0000 7f5a74641e40 20 remove_watcher() i=6 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.802+0000 7f5a74641e40 20 remove_watcher() i=7 2023-10-05T19:35:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.806+0000 7f5a43fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:49.837 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-10-05T19:35:49.837 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key AGVUHNNTZZVPWUCVHGBJ --secret nEEO3GWeSvX2kLazegQLNO8w7GgEPDJ3NmaMLMh0FLGEzDV2Ff+unA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:35:49.883 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:50.068 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.890+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 realm 2023-10-05T19:35:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.894+0000 7f3da38e0e40 20 searching for the correct realm 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.918+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.918+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.918+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.918+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.918+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.922+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.946+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.946+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.946+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.946+0000 7f3da38e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.946+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 zone default found 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 4 Realm: () 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:50.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 zonegroup default 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.950+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.954+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.954+0000 7f3da38e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.954+0000 7f3da38e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.954+0000 7f3da38e0e40 20 started sync module instance, tier type = 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.954+0000 7f3da38e0e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=0 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=1 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=2 2023-10-05T19:35:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=3 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=4 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=5 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=6 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 20 add_watcher() i=7 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.978+0000 7f3da38e0e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.982+0000 7f3d727e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.982+0000 7f3da38e0e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.982+0000 7f3da38e0e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.990+0000 7f3da38e0e40 5 note: GC not initialized 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.990+0000 7f3ce17fa700 20 reqs_thread_entry: start 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.990+0000 7f3cdaffd700 20 reqs_thread_entry: start 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.990+0000 7f3da38e0e40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:50.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3cd97fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:20 2023 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3cd8ff9700 20 reqs_thread_entry: start 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 20 get_system_obj_state: rctx=0x7ffd6ab3a1f8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55f3820520b0 s->prefetch_data=0 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 20 get_system_obj_state: rctx=0x7ffd6ab39ee8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55f3820520b0 s->prefetch_data=0 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 20 get_system_obj_state: rctx=0x7ffd6ab39ee8 obj=default.rgw.meta:users.keys:AGVUHNNTZZVPWUCVHGBJ state=0x55f3820520b0 s->prefetch_data=0 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache get: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ : miss 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ info.flags=0x0 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 adding default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ to cache LRU end 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 20 get_system_obj_state: rctx=0x7ffd6ab39788 obj=default.rgw.meta:users.keys:AGVUHNNTZZVPWUCVHGBJ state=0x55f3820520b0 s->prefetch_data=0 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache get: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ : hit (negative entry) 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 20 get_system_obj_state: rctx=0x7ffd6ab39e28 obj=default.rgw.meta:users.keys:AGVUHNNTZZVPWUCVHGBJ state=0x55f3820520b0 s->prefetch_data=0 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.994+0000 7f3da38e0e40 10 cache get: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ : hit (negative entry) 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.998+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.998+0000 7f3da38e0e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:49.998+0000 7f3da38e0e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:35:50.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3d9a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94504348248672 notifier 4223 bl.length()=640 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3d9a7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3d9a7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3da38e0e40 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.002+0000 7f3da38e0e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.006+0000 7f3d9a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94504348248672 notifier 4223 bl.length()=194 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.006+0000 7f3d9a7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.006+0000 7f3d9a7fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3da38e0e40 10 cache put: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ info.flags=0x7 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3da38e0e40 10 moving default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ to cache LRU end 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3da38e0e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:AGVUHNNTZZVPWUCVHGBJ, ofs0, ns] 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3d9affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018880 cookie 94504348257072 notifier 4223 bl.length()=186 2023-10-05T19:35:50.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3d9affd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ info.flags=0x7 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.010+0000 7f3d9affd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+AGVUHNNTZZVPWUCVHGBJ to cache LRU end 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "user_id": "bar.client.0", 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "display_name": "Mr. bar.client.0", 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "email": "bar.client.0+test@test.test", 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "suspended": 0, 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max_buckets": 1000, 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "subusers": [], 2023-10-05T19:35:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "keys": [ 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "user": "bar.client.0", 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "access_key": "AGVUHNNTZZVPWUCVHGBJ", 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "secret_key": "nEEO3GWeSvX2kLazegQLNO8w7GgEPDJ3NmaMLMh0FLGEzDV2Ff+unA==" 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "swift_keys": [], 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": [ 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "user-policy", 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "perm": "*" 2023-10-05T19:35:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "op_mask": "read, write, delete", 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "default_placement": "", 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "default_storage_class": "", 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_tags": [], 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_quota": { 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "user_quota": { 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "temp_url_keys": [], 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "rgw", 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "mfa_ids": [] 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-05T19:35:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.014+0000 7f3da38e0e40 20 remove_watcher() i=0 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.014+0000 7f3da38e0e40 2 removed watcher, disabling cache 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.014+0000 7f3da38e0e40 20 remove_watcher() i=1 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.018+0000 7f3da38e0e40 20 remove_watcher() i=2 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.018+0000 7f3da38e0e40 20 remove_watcher() i=3 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.022+0000 7f3da38e0e40 20 remove_watcher() i=4 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.022+0000 7f3da38e0e40 20 remove_watcher() i=5 2023-10-05T19:35:50.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.026+0000 7f3da38e0e40 20 remove_watcher() i=6 2023-10-05T19:35:50.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.026+0000 7f3da38e0e40 20 remove_watcher() i=7 2023-10-05T19:35:50.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.026+0000 7f3d727e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.090 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 8570576586 --totp-seed NX3BFMJSLFFXYXN3BLN6VTZUYG3Q72TTDC5XX2VOYYKRPTRWAFWDHD3IUAZVH3TD --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:35:50.136 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:50.136 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:50.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.142+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 realm 2023-10-05T19:35:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:50.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.146+0000 7fe14c543e40 20 searching for the correct realm 2023-10-05T19:35:50.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.166+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.186+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 zone default found 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 4 Realm: () 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 zonegroup default 2023-10-05T19:35:50.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 started sync module instance, tier type = 2023-10-05T19:35:50.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.190+0000 7fe14c543e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:50.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=0 2023-10-05T19:35:50.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=1 2023-10-05T19:35:50.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=2 2023-10-05T19:35:50.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=3 2023-10-05T19:35:50.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=4 2023-10-05T19:35:50.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=5 2023-10-05T19:35:50.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=6 2023-10-05T19:35:50.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 20 add_watcher() i=7 2023-10-05T19:35:50.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.214+0000 7fe14c543e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.218+0000 7fe14c543e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.218+0000 7fe14c543e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.222+0000 7fe11b7e6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.226+0000 7fe14c543e40 5 note: GC not initialized 2023-10-05T19:35:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe08a7fc700 20 reqs_thread_entry: start 2023-10-05T19:35:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe07bfff700 20 reqs_thread_entry: start 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe07a7fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:20 2023 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe079ffb700 20 reqs_thread_entry: start 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 get_system_obj_state: rctx=0x7ffcd3e48828 obj=default.rgw.meta:users.uid:bar.client.0 state=0x5643b79db0b0 s->prefetch_data=0 2023-10-05T19:35:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:35:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:35:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T19:35:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.230+0000 7fe14c543e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:50.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe14c543e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:35:50.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe14c543e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe14c543e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:35:50.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe143fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94848844822960 notifier 4229 bl.length()=654 2023-10-05T19:35:50.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe143fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:35:50.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe143fff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:35:50.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe14c543e40 20 remove_watcher() i=0 2023-10-05T19:35:50.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.238+0000 7fe14c543e40 2 removed watcher, disabling cache 2023-10-05T19:35:50.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.242+0000 7fe14c543e40 20 remove_watcher() i=1 2023-10-05T19:35:50.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.242+0000 7fe14c543e40 20 remove_watcher() i=2 2023-10-05T19:35:50.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.246+0000 7fe14c543e40 20 remove_watcher() i=3 2023-10-05T19:35:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.250+0000 7fe14c543e40 20 remove_watcher() i=4 2023-10-05T19:35:50.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.250+0000 7fe14c543e40 20 remove_watcher() i=5 2023-10-05T19:35:50.258 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.254+0000 7fe14c543e40 20 remove_watcher() i=6 2023-10-05T19:35:50.259 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.254+0000 7fe14c543e40 20 remove_watcher() i=7 2023-10-05T19:35:50.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.258+0000 7fe11b7e6700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.270 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-10-05T19:35:50.271 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key WXHJGDAHUYMPQZHZNRMV --secret kTqSb45nfXM0RmCOUtBVK29woIqxeIcx8n0M3g7HFClQyd9W58FvBA== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-10-05T19:35:50.359 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:50.359 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:50.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.366+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 realm 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.370+0000 7fc310e4ce40 20 searching for the correct realm 2023-10-05T19:35:50.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.390+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.394+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.410+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 zone default found 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 4 Realm: () 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 zonegroup default 2023-10-05T19:35:50.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 started sync module instance, tier type = 2023-10-05T19:35:50.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.414+0000 7fc310e4ce40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=0 2023-10-05T19:35:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=1 2023-10-05T19:35:50.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=2 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=3 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=4 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=5 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.442+0000 7fc310e4ce40 20 add_watcher() i=6 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.446+0000 7fc310e4ce40 20 add_watcher() i=7 2023-10-05T19:35:50.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.446+0000 7fc310e4ce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:50.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.446+0000 7fc2dffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.446+0000 7fc310e4ce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:50.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.446+0000 7fc310e4ce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:50.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.454+0000 7fc310e4ce40 5 note: GC not initialized 2023-10-05T19:35:50.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc24effd700 20 reqs_thread_entry: start 2023-10-05T19:35:50.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc24cff9700 20 reqs_thread_entry: start 2023-10-05T19:35:50.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc242ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:20 2023 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc2427fc700 20 reqs_thread_entry: start 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 get_system_obj_state: rctx=0x7fffe7d492a8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x558a25ab70b0 s->prefetch_data=0 2023-10-05T19:35:50.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 get_system_obj_state: rctx=0x7fffe7d48f98 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x558a25ab70b0 s->prefetch_data=0 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 get_system_obj_state: rctx=0x7fffe7d48f98 obj=default.rgw.meta:users.keys:WXHJGDAHUYMPQZHZNRMV state=0x558a25ab70b0 s->prefetch_data=0 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache get: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV : miss 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV info.flags=0x0 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 adding default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV to cache LRU end 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 get_system_obj_state: rctx=0x7fffe7d48838 obj=default.rgw.meta:users.keys:WXHJGDAHUYMPQZHZNRMV state=0x558a25ab70b0 s->prefetch_data=0 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache get: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV : hit (negative entry) 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 20 get_system_obj_state: rctx=0x7fffe7d48ed8 obj=default.rgw.meta:users.keys:WXHJGDAHUYMPQZHZNRMV state=0x558a25ab70b0 s->prefetch_data=0 2023-10-05T19:35:50.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.458+0000 7fc310e4ce40 10 cache get: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV : hit (negative entry) 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc310e4ce40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc310e4ce40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc2fffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94051827313584 notifier 4235 bl.length()=714 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc2fffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:35:50.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.462+0000 7fc2fffff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:50.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T19:35:50.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc30c97d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94051827310864 notifier 4235 bl.length()=224 2023-10-05T19:35:50.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc30c97d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:35:50.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc30c97d700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:35:50.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 cache put: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV info.flags=0x7 2023-10-05T19:35:50.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 moving default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV to cache LRU end 2023-10-05T19:35:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.466+0000 7fc310e4ce40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:WXHJGDAHUYMPQZHZNRMV, ofs0, ns] 2023-10-05T19:35:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.470+0000 7fc2fffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94051827313584 notifier 4235 bl.length()=201 2023-10-05T19:35:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.470+0000 7fc2fffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV info.flags=0x7 2023-10-05T19:35:50.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.470+0000 7fc2fffff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+WXHJGDAHUYMPQZHZNRMV to cache LRU end 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "user_id": "testx$tenanteduser.client.0", 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "suspended": 0, 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "max_buckets": 1000, 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "subusers": [], 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "keys": [ 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "user": "testx$tenanteduser.client.0", 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "access_key": "WXHJGDAHUYMPQZHZNRMV", 2023-10-05T19:35:50.475 INFO:teuthology.orchestra.run.smithi064.stdout: "secret_key": "kTqSb45nfXM0RmCOUtBVK29woIqxeIcx8n0M3g7HFClQyd9W58FvBA==" 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "swift_keys": [], 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": [ 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "user-policy", 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "perm": "*" 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "op_mask": "read, write, delete", 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "default_placement": "", 2023-10-05T19:35:50.476 INFO:teuthology.orchestra.run.smithi064.stdout: "default_storage_class": "", 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "placement_tags": [], 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "bucket_quota": { 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "user_quota": { 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "enabled": false, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "check_on_raw": false, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size": -1, 2023-10-05T19:35:50.477 INFO:teuthology.orchestra.run.smithi064.stdout: "max_size_kb": 0, 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: "max_objects": -1 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: "temp_url_keys": [], 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "rgw", 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: "mfa_ids": [] 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-05T19:35:50.478 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:35:50.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.470+0000 7fc310e4ce40 20 remove_watcher() i=0 2023-10-05T19:35:50.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.470+0000 7fc310e4ce40 2 removed watcher, disabling cache 2023-10-05T19:35:50.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.474+0000 7fc310e4ce40 20 remove_watcher() i=1 2023-10-05T19:35:50.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.474+0000 7fc310e4ce40 20 remove_watcher() i=2 2023-10-05T19:35:50.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.474+0000 7fc310e4ce40 20 remove_watcher() i=3 2023-10-05T19:35:50.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.478+0000 7fc310e4ce40 20 remove_watcher() i=4 2023-10-05T19:35:50.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.478+0000 7fc310e4ce40 20 remove_watcher() i=5 2023-10-05T19:35:50.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.478+0000 7fc310e4ce40 20 remove_watcher() i=6 2023-10-05T19:35:50.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.482+0000 7fc310e4ce40 20 remove_watcher() i=7 2023-10-05T19:35:50.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.482+0000 7fc2dffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.494 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 6173455024 --totp-seed E3IS7LMRTCQLCSBQOM5KLPO57CV367VMSLDZFH7WPI75F4WS7BD2J5JK3YVDEGMF --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:35:50.540 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:35:50.540 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:35:50.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 realm 2023-10-05T19:35:50.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.550+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.554+0000 7f26ea107e40 20 searching for the correct realm 2023-10-05T19:35:50.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.574+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.578+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.578+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:35:50.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:35:50.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:35:50.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:35:50.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.598+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:35:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:35:50.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:35:50.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 zone default found 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 4 Realm: () 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:35:50.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:35:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 zonegroup default 2023-10-05T19:35:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.602+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.606+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.606+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.606+0000 7f26ea107e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:35:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.606+0000 7f26ea107e40 20 started sync module instance, tier type = 2023-10-05T19:35:50.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.606+0000 7f26ea107e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:35:50.628 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=0 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=1 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=2 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=3 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=4 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.622+0000 7f26ea107e40 20 add_watcher() i=5 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26ea107e40 20 add_watcher() i=6 2023-10-05T19:35:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26ea107e40 20 add_watcher() i=7 2023-10-05T19:35:50.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26ea107e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:35:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26b97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26ea107e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:35:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.626+0000 7f26ea107e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:35:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f26ea107e40 5 note: GC not initialized 2023-10-05T19:35:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f2625ffb700 20 reqs_thread_entry: start 2023-10-05T19:35:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f26ea107e40 20 init_complete bucket index max shards: 11 2023-10-05T19:35:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f26ea107e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:35:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f261bfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:36:20 2023 2023-10-05T19:35:50.640 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f261b7fe700 20 reqs_thread_entry: start 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f26ea107e40 20 get_system_obj_state: rctx=0x7ffe2f832a68 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55f9bf4950b0 s->prefetch_data=0 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.634+0000 7f26ea107e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f2627fff700 20 reqs_thread_entry: start 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 20 rados->read ofs=0 len=0 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 20 rados_obj.operate() r=0 bl.length=514 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.638+0000 7f26ea107e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:35:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26ea107e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:35:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26ea107e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26ea107e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:35:50.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26e5c38700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94531145929552 notifier 4241 bl.length()=728 2023-10-05T19:35:50.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26e5c38700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:35:50.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.642+0000 7f26e5c38700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:35:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.646+0000 7f26ea107e40 20 remove_watcher() i=0 2023-10-05T19:35:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.646+0000 7f26ea107e40 2 removed watcher, disabling cache 2023-10-05T19:35:50.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.650+0000 7f26ea107e40 20 remove_watcher() i=1 2023-10-05T19:35:50.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.650+0000 7f26ea107e40 20 remove_watcher() i=2 2023-10-05T19:35:50.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.654+0000 7f26ea107e40 20 remove_watcher() i=3 2023-10-05T19:35:50.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.654+0000 7f26ea107e40 20 remove_watcher() i=4 2023-10-05T19:35:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.658+0000 7f26ea107e40 20 remove_watcher() i=5 2023-10-05T19:35:50.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.658+0000 7f26ea107e40 20 remove_watcher() i=6 2023-10-05T19:35:50.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.662+0000 7f26ea107e40 20 remove_watcher() i=7 2023-10-05T19:35:50.667 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:35:50.662+0000 7f26b97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:35:50.677 INFO:tasks.s3tests:Configuring s3-tests... 2023-10-05T19:35:50.677 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2023-10-05T19:35:50.847 INFO:teuthology.orchestra.run.smithi064.stderr:./bootstrap: missing required DEB packages. Installing via sudo. 2023-10-05T19:35:50.913 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:35:51.103 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:35:51.104 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:35:51.282 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:35:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:35:51.283 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:35:51.284 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-05T19:35:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2 libxslt1.1 2023-10-05T19:35:51.285 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-05T19:35:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: pkg-config 2023-10-05T19:35:51.320 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-05T19:35:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2-dev 2023-10-05T19:35:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: libxslt1-dev 2023-10-05T19:35:51.323 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-05T19:35:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: libxml2 libxslt1.1 2023-10-05T19:35:51.380 INFO:teuthology.orchestra.run.smithi064.stdout:2 upgraded, 5 newly installed, 0 to remove and 249 not upgraded. 2023-10-05T19:35:51.380 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 2,082 kB of archives. 2023-10-05T19:35:51.380 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 7,578 kB of additional disk space will be used. 2023-10-05T19:35:51.381 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2 amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [640 kB] 2023-10-05T19:35:51.518 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-extra-2.1-7 amd64 2.1.11-stable-1 [60.0 kB] 2023-10-05T19:35:51.520 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-openssl-2.1-7 amd64 2.1.11-stable-1 [14.3 kB] 2023-10-05T19:35:51.521 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-dev amd64 2.1.11-stable-1 [261 kB] 2023-10-05T19:35:51.527 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2-dev amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [735 kB] 2023-10-05T19:35:51.547 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1.1 amd64 1.1.34-4ubuntu0.20.04.1 [151 kB] 2023-10-05T19:35:51.551 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1-dev amd64 1.1.34-4ubuntu0.20.04.1 [219 kB] 2023-10-05T19:35:51.823 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: unable to initialize frontend: Dialog 2023-10-05T19:35:51.823 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2023-10-05T19:35:51.824 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: falling back to frontend: Readline 2023-10-05T19:35:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: unable to initialize frontend: Readline 2023-10-05T19:35:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: (This frontend requires a controlling tty.) 2023-10-05T19:35:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:debconf: falling back to frontend: Teletype 2023-10-05T19:35:51.857 INFO:teuthology.orchestra.run.smithi064.stderr:dpkg-preconfigure: unable to re-open stdin: 2023-10-05T19:35:51.910 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 2,082 kB in 0s (9,273 kB/s) 2023-10-05T19:35:52.135 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152752 files and directories currently installed.) 2023-10-05T19:35:52.139 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-libxml2_2.9.10+dfsg-5ubuntu0.20.04.6_amd64.deb ... 2023-10-05T19:35:52.479 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) over (2.9.10+dfsg-5ubuntu0.20.04.4) ... 2023-10-05T19:35:52.950 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libevent-extra-2.1-7:amd64. 2023-10-05T19:35:52.967 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-libevent-extra-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-10-05T19:35:53.033 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-10-05T19:35:53.369 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libevent-openssl-2.1-7:amd64. 2023-10-05T19:35:53.388 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-libevent-openssl-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-10-05T19:35:53.436 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-10-05T19:35:53.747 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libevent-dev. 2023-10-05T19:35:53.764 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-libevent-dev_2.1.11-stable-1_amd64.deb ... 2023-10-05T19:35:53.805 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libevent-dev (2.1.11-stable-1) ... 2023-10-05T19:35:54.159 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libxml2-dev:amd64. 2023-10-05T19:35:54.179 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-libxml2-dev_2.9.10+dfsg-5ubuntu0.20.04.6_amd64.deb ... 2023-10-05T19:35:54.250 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-10-05T19:35:54.816 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-libxslt1.1_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:35:54.946 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) over (1.1.34-4) ... 2023-10-05T19:35:55.726 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libxslt1-dev:amd64. 2023-10-05T19:35:55.744 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-libxslt1-dev_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:35:55.809 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-10-05T19:35:56.220 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-10-05T19:35:56.373 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-10-05T19:35:56.482 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libevent-dev (2.1.11-stable-1) ... 2023-10-05T19:35:56.608 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-10-05T19:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-10-05T19:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-10-05T19:35:57.053 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-10-05T19:35:57.180 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:35:58.883 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:35:59.953 INFO:teuthology.orchestra.run.smithi064.stdout:created virtual environment CPython3.8.10.final.0-64 in 315ms 2023-10-05T19:35:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: creator CPython3Posix(dest=/home/ubuntu/cephtest/s3-tests-client.0/virtualenv, clear=False, global=False) 2023-10-05T19:35:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1) 2023-10-05T19:35:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator 2023-10-05T19:36:01.256 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pip 2023-10-05T19:36:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached pip-23.2.1-py3-none-any.whl (2.1 MB) 2023-10-05T19:36:01.367 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: pip 2023-10-05T19:36:01.368 INFO:teuthology.orchestra.run.smithi064.stdout: Attempting uninstall: pip 2023-10-05T19:36:01.368 INFO:teuthology.orchestra.run.smithi064.stdout: Found existing installation: pip 20.0.2 2023-10-05T19:36:01.418 INFO:teuthology.orchestra.run.smithi064.stdout: Uninstalling pip-20.0.2: 2023-10-05T19:36:01.421 INFO:teuthology.orchestra.run.smithi064.stdout: Successfully uninstalled pip-20.0.2 2023-10-05T19:36:02.357 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed pip-23.2.1 2023-10-05T19:36:03.106 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting setuptools==44.1.0 2023-10-05T19:36:03.188 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-10-05T19:36:03.293 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 583.4/583.4 kB 5.8 MB/s eta 0:00:00 2023-10-05T19:36:03.312 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: setuptools 2023-10-05T19:36:03.312 INFO:teuthology.orchestra.run.smithi064.stdout: Attempting uninstall: setuptools 2023-10-05T19:36:03.313 INFO:teuthology.orchestra.run.smithi064.stdout: Found existing installation: setuptools 44.0.0 2023-10-05T19:36:03.321 INFO:teuthology.orchestra.run.smithi064.stdout: Uninstalling setuptools-44.0.0: 2023-10-05T19:36:03.328 INFO:teuthology.orchestra.run.smithi064.stdout: Successfully uninstalled setuptools-44.0.0 2023-10-05T19:36:03.565 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed setuptools-44.1.0 2023-10-05T19:36:04.383 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting PyYAML (from -r requirements.txt (line 1)) 2023-10-05T19:36:04.383 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for PyYAML from https://files.pythonhosted.org/packages/c8/6b/6600ac24725c7388255b2f5add93f91e58a5d7efaf4af244fdbcc11a541b/PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:36:04.449 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) 2023-10-05T19:36:04.551 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting nose>=1.0.0 (from -r requirements.txt (line 2)) 2023-10-05T19:36:04.570 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-10-05T19:36:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 154.7/154.7 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:36:04.736 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting boto>=2.6.0 (from -r requirements.txt (line 3)) 2023-10-05T19:36:04.759 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-10-05T19:36:04.840 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 1.4/1.4 MB 17.7 MB/s eta 0:00:00 2023-10-05T19:36:05.492 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 4)) 2023-10-05T19:36:05.492 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/4d/65/a0927694bb02d8ae46c399d9a8ed2cd9c724adbc9bbb7849a0650850ad31/boto3-1.28.61-py3-none-any.whl.metadata 2023-10-05T19:36:05.510 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.61-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:06.209 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting botocore<1.28.0 (from -r requirements.txt (line 6)) 2023-10-05T19:36:06.230 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading botocore-1.27.96-py3-none-any.whl (9.3 MB) 2023-10-05T19:36:06.438 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 9.3/9.3 MB 45.6 MB/s eta 0:00:00 2023-10-05T19:36:06.539 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting munch>=2.0.0 (from -r requirements.txt (line 7)) 2023-10-05T19:36:06.540 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for munch>=2.0.0 from https://files.pythonhosted.org/packages/56/b3/7c69b37f03260a061883bec0e7b05be7117c1b1c85f5212c72c8c2bc3c8c/munch-4.0.0-py2.py3-none-any.whl.metadata 2023-10-05T19:36:06.558 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl.metadata (5.9 kB) 2023-10-05T19:36:06.851 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting gevent>=1.0 (from -r requirements.txt (line 9)) 2023-10-05T19:36:06.851 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for gevent>=1.0 from https://files.pythonhosted.org/packages/65/f9/06b126756581156693772929c9a211688a07e799850f47e858e2f084fb8a/gevent-23.9.1-cp38-cp38-manylinux_2_28_x86_64.whl.metadata 2023-10-05T19:36:06.876 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading gevent-23.9.1-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (13 kB) 2023-10-05T19:36:06.982 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting isodate>=0.4.4 (from -r requirements.txt (line 10)) 2023-10-05T19:36:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-10-05T19:36:07.029 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 41.7/41.7 kB 1.3 MB/s eta 0:00:00 2023-10-05T19:36:07.123 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting requests>=2.23.0 (from -r requirements.txt (line 11)) 2023-10-05T19:36:07.123 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for requests>=2.23.0 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata 2023-10-05T19:36:07.124 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) 2023-10-05T19:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pytz>=2011k (from -r requirements.txt (line 12)) 2023-10-05T19:36:07.233 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for pytz>=2011k from https://files.pythonhosted.org/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl.metadata 2023-10-05T19:36:07.237 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB) 2023-10-05T19:36:07.308 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting httplib2 (from -r requirements.txt (line 13)) 2023-10-05T19:36:07.328 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-10-05T19:36:07.364 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 96.9/96.9 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:07.661 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting lxml (from -r requirements.txt (line 14)) 2023-10-05T19:36:07.661 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for lxml from https://files.pythonhosted.org/packages/44/1b/0771c38e65ad23e25368b5e07c920054774b8d12477a4fad116bf500de73/lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl.metadata 2023-10-05T19:36:07.685 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (3.8 kB) 2023-10-05T19:36:07.831 INFO:teuthology.orchestra.run.smithi064.stdout:INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while. 2023-10-05T19:36:07.835 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 4)) 2023-10-05T19:36:07.835 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/da/7e/ea3b898a2f7f8b698eb93d808cca4202e2e5ee1887715b28b42fb9b5c71c/boto3-1.28.60-py3-none-any.whl.metadata 2023-10-05T19:36:07.868 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.60-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/88/a3/e63a11c957c1e2a901d13f3ddc2acba2d0db8a7bcbac2e3f5b06aab0c6de/boto3-1.28.59-py3-none-any.whl.metadata 2023-10-05T19:36:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.59-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:08.259 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/90/ca/a9e28e3d0c924743d8086de1ba9afae1406ac2811cf809abac5c69467a57/boto3-1.28.58-py3-none-any.whl.metadata 2023-10-05T19:36:08.277 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.58-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:08.513 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/91/a9/c6eea6c80c57d8957eaba3cc1122339a1adbdce60532cbf94046c312bcd3/boto3-1.28.57-py3-none-any.whl.metadata 2023-10-05T19:36:08.531 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.57-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/9b/86/1c5a07d3c93e80b22e6dce8c2280199e6e12dbf72dd8738d4f5e65887bc8/boto3-1.28.56-py3-none-any.whl.metadata 2023-10-05T19:36:08.769 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.56-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:08.956 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/1e/3e/827f665b39f95c74d040d4745fdbb9f1097e40ef705fac2ef0eaa3f63f6a/boto3-1.28.55-py3-none-any.whl.metadata 2023-10-05T19:36:08.973 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.55-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:09.114 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/60/d0/9ce6abb39e0c037028172ad3465e76ac9c8ed39095028109fe7c54e8aa28/boto3-1.28.54-py3-none-any.whl.metadata 2023-10-05T19:36:09.133 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.54-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:09.371 INFO:teuthology.orchestra.run.smithi064.stdout:INFO: pip is still looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while. 2023-10-05T19:36:09.376 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/fe/15/fa88dc3bf239fe047fcd9f7c1c38b655f533d453d4d6c3f72ab5b145d44d/boto3-1.28.53-py3-none-any.whl.metadata 2023-10-05T19:36:09.400 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.53-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:09.589 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/7e/47/7613bc648576e5be0da6c7feb2a5cfba1f6b36e7b163adbdc5ce4ebd8f16/boto3-1.28.52-py3-none-any.whl.metadata 2023-10-05T19:36:09.613 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.52-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:09.751 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/fe/19/6699107cf1cad26b5ba4aa6206037151e30ab5673f0c5041de6ddfacbbe5/boto3-1.28.51-py3-none-any.whl.metadata 2023-10-05T19:36:09.769 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.51-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:09.985 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/e7/17/8a58bb2de10d19333d4aae8a9719191899da956e894ffa2caa4eb9f2903b/boto3-1.28.50-py3-none-any.whl.metadata 2023-10-05T19:36:10.005 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.50-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:10.154 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/84/85/e895e6ca369f406432c5f5f186b107f1e1a6a2408b60a081de32021fd818/boto3-1.28.49-py3-none-any.whl.metadata 2023-10-05T19:36:10.172 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.49-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:10.309 INFO:teuthology.orchestra.run.smithi064.stdout:INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. 2023-10-05T19:36:10.313 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/5c/75/e70d030dc583cd76f1168fbb5da58e930dafcc947f3d4c2e8cdae68b9459/boto3-1.28.48-py3-none-any.whl.metadata 2023-10-05T19:36:10.331 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.48-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/bf/b1/8c254dbb9e315f3b2f5a2a30667a08aed13fd9a00e07028eee701c7ce00d/boto3-1.28.47-py3-none-any.whl.metadata 2023-10-05T19:36:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.47-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:10.629 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/7f/34/c2bcec634de677a7d05a868e72fd945bd40a2faf2ce249d773bd96c7eec2/boto3-1.28.46-py3-none-any.whl.metadata 2023-10-05T19:36:10.647 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.46-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:10.903 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/10/b1/589faa016ad2d76809e9bed20cca03784248fdce74a9edd1d9eec4b1d454/boto3-1.28.45-py3-none-any.whl.metadata 2023-10-05T19:36:10.922 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.45-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:11.140 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/85/aa/7f8313a310325d9c1ef0b8c34295018637ed4989bdef13c2831758561780/boto3-1.28.44-py3-none-any.whl.metadata 2023-10-05T19:36:11.158 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.44-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:11.299 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6e/7f/ffb72ddc9f465183af04623baf9d0ea70e73cb0957407e4c333b9e0263fb/boto3-1.28.43-py3-none-any.whl.metadata 2023-10-05T19:36:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.43-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:11.555 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/af/9c/58ca26190a8b89776ec66fdf769d6d0d5287044417b58df4d8225a081153/boto3-1.28.42-py3-none-any.whl.metadata 2023-10-05T19:36:11.574 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.42-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:11.712 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/7e/3a/eb1ec341a1028ba5894924e1e3011116c7a7736060ac5f7652de1a0b187b/boto3-1.28.41-py3-none-any.whl.metadata 2023-10-05T19:36:11.730 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.41-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:11.869 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/81/09/be4b462f1aef33ffd789dbcfbd626a261fa0acc3b1499990ce8b71474d89/boto3-1.28.40-py3-none-any.whl.metadata 2023-10-05T19:36:11.888 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.40-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6a/22/a9272e54f32aedf82c6b27f989765025365c2a529516bb60a0b11f0db88c/boto3-1.28.39-py3-none-any.whl.metadata 2023-10-05T19:36:12.064 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.39-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/b0/ab/d412e2874631b0d8b2de5b20526ed5ef63d38a0d16cbe0ea235cf98f7bc0/boto3-1.28.38-py3-none-any.whl.metadata 2023-10-05T19:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.38-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/be/ed/c65f67d71526fa7cf2f9d09ea5b3c81ad3eba9c9e324fabdffdad13f00de/boto3-1.28.37-py3-none-any.whl.metadata 2023-10-05T19:36:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.37-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:12.685 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/13/ba/439870e6f75a49cd6425146b70557ea0154643ee32b282de29e4c72087e7/boto3-1.28.36-py3-none-any.whl.metadata 2023-10-05T19:36:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.36-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:12.851 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/c7/e7/5c557d60eb76ba55ff202bea08e15d1840ba0fe8d1c7bfccf54b8c5a2990/boto3-1.28.35-py3-none-any.whl.metadata 2023-10-05T19:36:12.869 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.35-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/d5/f6/0bdda8eefd2151f35bd6aa6ada5f3494b0baf83f0be874a82a069168115e/boto3-1.28.34-py3-none-any.whl.metadata 2023-10-05T19:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.34-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:13.329 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/8c/64/bdc3f64273d27f27bf0a082de64be341109b4ce664d95ebd77c16cfd40b2/boto3-1.28.33-py3-none-any.whl.metadata 2023-10-05T19:36:13.347 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.33-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:13.493 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/65/cf/8962dcee7c0985d542f0e6c5040de91a394038be9e199c82f247df16ecbc/boto3-1.28.32-py3-none-any.whl.metadata 2023-10-05T19:36:13.522 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.32-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/60/6d/8e44bad2e87b30396093f30e22e44ea702a8ef5dc0c2cff60632f1d54cbe/boto3-1.28.31-py3-none-any.whl.metadata 2023-10-05T19:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.31-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/7c/f5/6b4302a6d34715d47f7d729e13cae7a2b896b7272561b1b1d4b714a0b76b/boto3-1.28.30-py3-none-any.whl.metadata 2023-10-05T19:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.30-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:14.114 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/81/c1/29c75866b41bda4bfca883e2306cda8ac8c02eb0ecb26e1225b091125949/boto3-1.28.29-py3-none-any.whl.metadata 2023-10-05T19:36:14.132 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.29-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:14.349 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6a/4d/80f7b7d6fa0a25def2d38d3161f2d3b2af56bb8619824182488c8225ff23/boto3-1.28.28-py3-none-any.whl.metadata 2023-10-05T19:36:14.367 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.28-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:14.563 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/ec/9a/c0837684f1ab666add90e639944575f7325301d59d19f72b6acf6c850b78/boto3-1.28.27-py3-none-any.whl.metadata 2023-10-05T19:36:14.581 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.27-py3-none-any.whl.metadata (6.7 kB) 2023-10-05T19:36:14.810 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/11/c1/6ee53bbb333a7097281592d39be2d70c0bb65c28ce969004a26ce68df9ce/boto3-1.28.26-py3-none-any.whl.metadata 2023-10-05T19:36:14.828 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.26-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/d2/c9/9af79d74ec14c3a5a69c08c23ad4e30d752826b89399aeb5afff8e18515e/boto3-1.28.25-py3-none-any.whl.metadata 2023-10-05T19:36:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.25-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:15.223 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/db/4c/f17ce85f01b7782e15a09ddc2a1d72952a08c685f91501a212e0c9390e50/boto3-1.28.24-py3-none-any.whl.metadata 2023-10-05T19:36:15.242 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.24-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/24/35/37578a6f429ccf41717207ed7b0805bf3cec71ebc038a4ec036bc36432b4/boto3-1.28.23-py3-none-any.whl.metadata 2023-10-05T19:36:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.23-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:15.633 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/2f/83/07bbd31d54f8cfbb30cd09802c2eced29a50324f746520eefd69cc7002f6/boto3-1.28.22-py3-none-any.whl.metadata 2023-10-05T19:36:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.22-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/20/eb/85a7acc167a31796961299b0444ab0f95798b38f173b5f264c51b483beb8/boto3-1.28.21-py3-none-any.whl.metadata 2023-10-05T19:36:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.21-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:16.005 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/cd/5b/992af4c79af98a1821bcedad6a74b41e373c7cd86893833a54b1472d43fd/boto3-1.28.20-py3-none-any.whl.metadata 2023-10-05T19:36:16.023 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.20-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:16.162 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/51/85/48dc639c57cb1a25b1f7b3022d2ab982fd97156715cb11f50102cb5c551f/boto3-1.28.19-py3-none-any.whl.metadata 2023-10-05T19:36:16.180 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.19-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:16.398 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/63/83/7c6e727e10bb7635aba98c42990d16921b7465fed3ec6606dc5622017b59/boto3-1.28.18-py3-none-any.whl.metadata 2023-10-05T19:36:16.416 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.18-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:16.652 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/46/a7/487512e3328f2566d72aed3b7059fd8dff18c95d9bcbbe16c5ecc13e6fc5/boto3-1.28.17-py3-none-any.whl.metadata 2023-10-05T19:36:16.678 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.17-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/e3/9c/5f339f649a888d7d016d505dd5a47c3d47ec0e550340e4e932a5718d549e/boto3-1.28.16-py3-none-any.whl.metadata 2023-10-05T19:36:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.16-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.036 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/f0/a0/2f034b2f961c44e48de230ee574327e3c835766f2209f9415aa66f7b41c5/boto3-1.28.15-py3-none-any.whl.metadata 2023-10-05T19:36:17.056 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.15-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.198 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/ce/4c/ded61d276a06360afaf8912b05c9e6124173f86208085edbf3ba234ab9ab/boto3-1.28.14-py3-none-any.whl.metadata 2023-10-05T19:36:17.217 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.14-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6a/87/9d2a8549e9237fcf791bac40dad0180a78052c41195db59c077b29614070/boto3-1.28.13-py3-none-any.whl.metadata 2023-10-05T19:36:17.373 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.13-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.537 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/3f/a1/83dfdfbb0e8240bc50d0e73dc6bbb47edcfaa495c1b2a584f7cc30d6b8a0/boto3-1.28.12-py3-none-any.whl.metadata 2023-10-05T19:36:17.632 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.12-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.697 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/9a/ce/5cbaf62a9c496d6c54bdd7ec8cf8cf6b1c37f4743dc781f88ef9b46389a7/boto3-1.28.11-py3-none-any.whl.metadata 2023-10-05T19:36:17.715 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.11-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:17.862 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/8d/c0/6a10c61ac2493503f8c28566d6203f444a0499a574c19c4feddc029e2f5c/boto3-1.28.10-py3-none-any.whl.metadata 2023-10-05T19:36:17.879 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.10-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:18.072 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/0b/5e/a219cc9aa913a0e6f203c27e42eeb600d27e24cf9dd105daac8daaea01e5/boto3-1.28.9-py3-none-any.whl.metadata 2023-10-05T19:36:18.090 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.9-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:18.234 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/9c/34/3aa6f19eca18bae15362ce747b01164191432947040d51c2ed45db803051/boto3-1.28.8-py3-none-any.whl.metadata 2023-10-05T19:36:18.252 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.8-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:18.408 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/d0/39/32c3b6f7b94f034fff0eda2ddcb5a106f69d3aa37df6b8f609d10d881003/boto3-1.28.7-py3-none-any.whl.metadata 2023-10-05T19:36:18.426 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.7-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:18.626 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/27/a6/eb9b9a16b8fe33257fd1546ea85eed59c152af088220a9e300a0763ab987/boto3-1.28.6-py3-none-any.whl.metadata 2023-10-05T19:36:18.645 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.6-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:18.792 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/0a/5c/055aa43caaad7e75a481ec8f3d74435d6e248d8171ee14ea826f34eecbd4/boto3-1.28.5-py3-none-any.whl.metadata 2023-10-05T19:36:18.809 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.5-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/e3/ac/ad93e6b6c958351b9314480d6b352ffb2da45087d4c48f52ff97cb3a3d39/boto3-1.28.4-py3-none-any.whl.metadata 2023-10-05T19:36:19.086 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.4-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.260 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/23/87/92592a5aa3eafcebd37117d4d03eef8fd922184813532f743ca607bfb28c/boto3-1.28.3-py3-none-any.whl.metadata 2023-10-05T19:36:19.278 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.3-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.418 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/23/88/a16cb7f0dc67501d1f0307097415e99516ef44236a369a248a3664f57849/boto3-1.28.2-py3-none-any.whl.metadata 2023-10-05T19:36:19.436 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.2-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.579 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/19/76/25af50b50fb67b275d8bfcba50243cf67964e95c52acbcf565d47471d43c/boto3-1.28.1-py3-none-any.whl.metadata 2023-10-05T19:36:19.597 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.1-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.727 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/58/9d/9b762f33b17a14359f1557e9a8d5db2887f5b040a47d9502de2ce4475cad/boto3-1.28.0-py3-none-any.whl.metadata 2023-10-05T19:36:19.750 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.28.0-py3-none-any.whl.metadata (6.6 kB) 2023-10-05T19:36:19.956 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6f/96/46b6e8ae69329cf121ee9ae6c9d59f7da6b74a90296e4927ca3fadf8ef1e/boto3-1.27.1-py3-none-any.whl.metadata 2023-10-05T19:36:19.974 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.27.1-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/0b/87/345b82f53e528614cb46ef412000a6adf1103cc103e396517a4f0d98ab84/boto3-1.27.0-py3-none-any.whl.metadata 2023-10-05T19:36:20.178 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.27.0-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:20.322 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/62/fa/04e99eb81c703e0ef6c8e503fc7d74c8d836157ed15d61bc77f4fd1c6e73/boto3-1.26.165-py3-none-any.whl.metadata 2023-10-05T19:36:20.341 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.165-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:20.525 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/9a/dc/fd034bc36d2fc0930ea7193dc4f1696a8748e4faefa51b07fb8deae64d2f/boto3-1.26.164-py3-none-any.whl.metadata 2023-10-05T19:36:20.545 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.164-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:20.751 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/2e/91/0894f0d826b46b76842274f04ce8b26be29a4dece6a851ef38883f991154/boto3-1.26.163-py3-none-any.whl.metadata 2023-10-05T19:36:20.769 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.163-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/38/cb/0de1d433d5d2dfb0adf022a74778c1f630d5c4ae846c093d65083b601ed2/boto3-1.26.162-py3-none-any.whl.metadata 2023-10-05T19:36:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.162-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/fb/2e/56a15ff0ab4518927bb909f1388ee062688cbde708fdbe52e9fb73ab3eae/boto3-1.26.161-py3-none-any.whl.metadata 2023-10-05T19:36:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.161-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:21.652 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/b2/c5/90a0417a7300ae070164ef6acfa1779d5534e9c1f257e28a0b57ffc762d8/boto3-1.26.160-py3-none-any.whl.metadata 2023-10-05T19:36:21.670 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.160-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:21.870 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/89/cf/94de9fc42e92ee49a18f32b91bc580b15de58cac6c45d57397ff9281f1fb/boto3-1.26.159-py3-none-any.whl.metadata 2023-10-05T19:36:21.889 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.159-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:22.083 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/04/05/48d42bd5ea19e0d979f348a744199452306333e4fc46e9eae17ab04b917e/boto3-1.26.158-py3-none-any.whl.metadata 2023-10-05T19:36:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.158-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:22.323 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/8d/b9/30b2716ab355e343e21d4e0969180b10b2682e81a3745821093923f22a8d/boto3-1.26.157-py3-none-any.whl.metadata 2023-10-05T19:36:22.373 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.157-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:22.593 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/ca/b0/3328333a30ca627f6e3619cb79caf4a1c889e7eb6bd9bf4d3755a3570d87/boto3-1.26.156-py3-none-any.whl.metadata 2023-10-05T19:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.156-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/e5/3b/919c4a90311d61615db459ac37721164528c7c829e27879b0927d0c0fc47/boto3-1.26.155-py3-none-any.whl.metadata 2023-10-05T19:36:22.854 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.155-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:23.004 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/0a/0f/b5771ff8ccd2f0baca29c218ebdd6a03129a79c476df8401df7621bbd255/boto3-1.26.154-py3-none-any.whl.metadata 2023-10-05T19:36:23.024 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.154-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:23.205 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/bd/62/2da5676690a67390879c280fe9b428c91d7f05fa7fb75918bcf24123b769/boto3-1.26.153-py3-none-any.whl.metadata 2023-10-05T19:36:23.222 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.153-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:23.407 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/8d/db/dc007fe486afa557289cdcda64f4669118c7769478a948939d11b3247e42/boto3-1.26.152-py3-none-any.whl.metadata 2023-10-05T19:36:23.425 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.152-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:23.618 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/60/07/c2575191ee209945cb542ff3ec0dd7382d1c656b35440df96c9b83cbb657/boto3-1.26.151-py3-none-any.whl.metadata 2023-10-05T19:36:23.636 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.151-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:23.819 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/37/13/cf36784c8c12cef44ef887f44042a7f8faeff684b469063e67f756911b1d/boto3-1.26.150-py3-none-any.whl.metadata 2023-10-05T19:36:23.837 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.150-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:24.046 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/1c/12/a747e5e24e6219ebee4162723c125083894ec71f4ee1308a6ed1560e4c94/boto3-1.26.149-py3-none-any.whl.metadata 2023-10-05T19:36:24.064 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.149-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:24.251 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/3b/3b/910ee6db909ff14aec58e568ff36128779c47600c2f58b8901f360565016/boto3-1.26.148-py3-none-any.whl.metadata 2023-10-05T19:36:24.270 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.148-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/3c/79/703467dca0345c351b2b38638e5c0838e76959209ec882581098ece5b3a3/boto3-1.26.147-py3-none-any.whl.metadata 2023-10-05T19:36:24.488 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.147-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:24.678 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/90/b0/bb112767cf6ae66ad57561ff3c6768257fd5c8d5d21b67385d743e060f0a/boto3-1.26.146-py3-none-any.whl.metadata 2023-10-05T19:36:24.696 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.146-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:24.912 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/dd/5f/d880fd9a9a5b3fec23dc07563ca9538d5f465dd4283331c588ca8fe0daab/boto3-1.26.145-py3-none-any.whl.metadata 2023-10-05T19:36:24.931 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.145-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:25.120 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/98/1e/d636fc4a63d9343a0e973e4b9e44b9a4d7a9531821bd8b3dc7fa720dc054/boto3-1.26.144-py3-none-any.whl.metadata 2023-10-05T19:36:25.139 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.144-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:25.332 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/41/2f/bec0be3471ad452f800c7540b864efd7ee080cacf35305b0efe60b52f880/boto3-1.26.143-py3-none-any.whl.metadata 2023-10-05T19:36:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.143-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:25.537 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/f8/fd/ccca7ea099b619c6951c6140ce381a2622f7033a8ac5613fc41d62394259/boto3-1.26.142-py3-none-any.whl.metadata 2023-10-05T19:36:25.555 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.142-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/56/0c/44d0f6b2f29590c6e9a2f1239640b67a49cec00dfbaeb6352be47c29661d/boto3-1.26.141-py3-none-any.whl.metadata 2023-10-05T19:36:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.141-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:25.991 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/b1/ea/821ed34ed6e901a37dcf37ca06009ed450544151b68dfbcc047a185a6d6a/boto3-1.26.140-py3-none-any.whl.metadata 2023-10-05T19:36:26.010 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.140-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:26.198 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/1c/33/c4301bb4fa0406e280bbf7eb36089200f9ad76b20b4674c5c3964177d1bf/boto3-1.26.139-py3-none-any.whl.metadata 2023-10-05T19:36:26.220 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.139-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6e/9c/407893f7829996a1a2f87825176acf20e1455699519b86ec9d3394146928/boto3-1.26.138-py3-none-any.whl.metadata 2023-10-05T19:36:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.138-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:26.706 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/20/8d/de46b65fccc863223d62ded9bd4b1e582036a2cb5658273644b9e7e4f582/boto3-1.26.137-py3-none-any.whl.metadata 2023-10-05T19:36:26.724 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.137-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:26.990 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/f0/7c/0bd42ba9e59e6e04fb243ae1b7b05f0e72cb23185a11686c95d0d61c7fc1/boto3-1.26.136-py3-none-any.whl.metadata 2023-10-05T19:36:27.009 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.136-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:27.246 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/6f/e1/36ddefb61f67619fb82d381899c0b3e8be5c65068d02cc8fadbbff14d1f5/boto3-1.26.135-py3-none-any.whl.metadata 2023-10-05T19:36:27.264 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.135-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:27.452 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for boto3>=1.0.0 from https://files.pythonhosted.org/packages/cf/60/4d74ca0d6a6ee907af6d4e3e734e745039ad12eaf833b74c7dde331566ac/boto3-1.26.134-py3-none-any.whl.metadata 2023-10-05T19:36:27.469 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.134-py3-none-any.whl.metadata (7.0 kB) 2023-10-05T19:36:27.675 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.133-py3-none-any.whl (135 kB) 2023-10-05T19:36:27.711 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:27.954 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.132-py3-none-any.whl (135 kB) 2023-10-05T19:36:27.997 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:28.177 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.131-py3-none-any.whl (135 kB) 2023-10-05T19:36:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:28.433 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.130-py3-none-any.whl (135 kB) 2023-10-05T19:36:28.473 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:28.652 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.129-py3-none-any.whl (135 kB) 2023-10-05T19:36:28.690 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.128-py3-none-any.whl (135 kB) 2023-10-05T19:36:28.924 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:29.102 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.127-py3-none-any.whl (135 kB) 2023-10-05T19:36:29.142 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:29.325 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.126-py3-none-any.whl (135 kB) 2023-10-05T19:36:29.358 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 4.5 MB/s eta 0:00:00 2023-10-05T19:36:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.125-py3-none-any.whl (135 kB) 2023-10-05T19:36:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.5 MB/s eta 0:00:00 2023-10-05T19:36:29.775 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.124-py3-none-any.whl (135 kB) 2023-10-05T19:36:29.841 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 2.0 MB/s eta 0:00:00 2023-10-05T19:36:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.123-py3-none-any.whl (135 kB) 2023-10-05T19:36:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.122-py3-none-any.whl (135 kB) 2023-10-05T19:36:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.121-py3-none-any.whl (135 kB) 2023-10-05T19:36:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 2.5 MB/s eta 0:00:00 2023-10-05T19:36:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.120-py3-none-any.whl (135 kB) 2023-10-05T19:36:30.869 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-10-05T19:36:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.119-py3-none-any.whl (135 kB) 2023-10-05T19:36:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:31.314 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.118-py3-none-any.whl (135 kB) 2023-10-05T19:36:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 1.1 MB/s eta 0:00:00 2023-10-05T19:36:31.613 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.117-py3-none-any.whl (135 kB) 2023-10-05T19:36:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: ------------------------------------- 135.6/135.6 kB 526.2 kB/s eta 0:00:00 2023-10-05T19:36:32.041 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.116-py3-none-any.whl (135 kB) 2023-10-05T19:36:32.080 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.115-py3-none-any.whl (135 kB) 2023-10-05T19:36:32.315 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.0 MB/s eta 0:00:00 2023-10-05T19:36:32.500 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.114-py3-none-any.whl (135 kB) 2023-10-05T19:36:32.541 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:32.724 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.113-py3-none-any.whl (135 kB) 2023-10-05T19:36:32.765 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:32.948 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.112-py3-none-any.whl (135 kB) 2023-10-05T19:36:32.982 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:33.158 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.111-py3-none-any.whl (135 kB) 2023-10-05T19:36:33.190 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:33.369 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.110-py3-none-any.whl (135 kB) 2023-10-05T19:36:33.417 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:33.589 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.109-py3-none-any.whl (135 kB) 2023-10-05T19:36:33.626 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:33.929 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.108-py3-none-any.whl (135 kB) 2023-10-05T19:36:33.967 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.107-py3-none-any.whl (135 kB) 2023-10-05T19:36:34.193 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.0 MB/s eta 0:00:00 2023-10-05T19:36:34.372 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.106-py3-none-any.whl (135 kB) 2023-10-05T19:36:34.411 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:34.601 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.105-py3-none-any.whl (135 kB) 2023-10-05T19:36:34.644 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.104-py3-none-any.whl (135 kB) 2023-10-05T19:36:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.103-py3-none-any.whl (135 kB) 2023-10-05T19:36:35.121 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:35.307 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.102-py3-none-any.whl (135 kB) 2023-10-05T19:36:35.355 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.6/135.6 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.101-py3-none-any.whl (135 kB) 2023-10-05T19:36:35.614 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 2.1 MB/s eta 0:00:00 2023-10-05T19:36:35.846 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.100-py3-none-any.whl (135 kB) 2023-10-05T19:36:35.872 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 5.8 MB/s eta 0:00:00 2023-10-05T19:36:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.99-py3-none-any.whl (135 kB) 2023-10-05T19:36:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 4.0 MB/s eta 0:00:00 2023-10-05T19:36:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.98-py3-none-any.whl (135 kB) 2023-10-05T19:36:36.307 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:36.481 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.97-py3-none-any.whl (135 kB) 2023-10-05T19:36:36.515 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.96-py3-none-any.whl (135 kB) 2023-10-05T19:36:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: ------------------------------------- 135.5/135.5 kB 826.3 kB/s eta 0:00:00 2023-10-05T19:36:37.078 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.95-py3-none-any.whl (135 kB) 2023-10-05T19:36:37.119 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.5/135.5 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:37.298 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.94-py3-none-any.whl (135 kB) 2023-10-05T19:36:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.1/135.1 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:37.514 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.93-py3-none-any.whl (135 kB) 2023-10-05T19:36:37.553 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 135.1/135.1 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:37.730 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.92-py3-none-any.whl (134 kB) 2023-10-05T19:36:37.770 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:37.956 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.91-py3-none-any.whl (134 kB) 2023-10-05T19:36:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:38.291 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.90-py3-none-any.whl (134 kB) 2023-10-05T19:36:38.331 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:38.513 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.89-py3-none-any.whl (134 kB) 2023-10-05T19:36:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:38.724 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.88-py3-none-any.whl (134 kB) 2023-10-05T19:36:38.758 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:38.940 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.87-py3-none-any.whl (134 kB) 2023-10-05T19:36:38.975 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 4.0 MB/s eta 0:00:00 2023-10-05T19:36:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.86-py3-none-any.whl (134 kB) 2023-10-05T19:36:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 4.2 MB/s eta 0:00:00 2023-10-05T19:36:39.405 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.85-py3-none-any.whl (134 kB) 2023-10-05T19:36:39.452 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:39.631 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.84-py3-none-any.whl (134 kB) 2023-10-05T19:36:39.691 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:36:39.911 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.83-py3-none-any.whl (134 kB) 2023-10-05T19:36:39.953 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:40.171 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.82-py3-none-any.whl (134 kB) 2023-10-05T19:36:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.5 MB/s eta 0:00:00 2023-10-05T19:36:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.81-py3-none-any.whl (134 kB) 2023-10-05T19:36:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 134.7/134.7 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:40.607 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.80-py3-none-any.whl (132 kB) 2023-10-05T19:36:40.645 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:40.822 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.79-py3-none-any.whl (132 kB) 2023-10-05T19:36:40.879 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:36:41.059 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.78-py3-none-any.whl (132 kB) 2023-10-05T19:36:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:41.280 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.77-py3-none-any.whl (132 kB) 2023-10-05T19:36:41.322 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:41.540 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.76-py3-none-any.whl (132 kB) 2023-10-05T19:36:41.580 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:41.759 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.75-py3-none-any.whl (132 kB) 2023-10-05T19:36:41.798 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:41.979 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.74-py3-none-any.whl (132 kB) 2023-10-05T19:36:42.041 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-10-05T19:36:42.221 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.73-py3-none-any.whl (132 kB) 2023-10-05T19:36:42.266 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.0 MB/s eta 0:00:00 2023-10-05T19:36:42.443 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.72-py3-none-any.whl (132 kB) 2023-10-05T19:36:42.483 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:42.659 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.71-py3-none-any.whl (132 kB) 2023-10-05T19:36:42.700 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:42.878 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.70-py3-none-any.whl (132 kB) 2023-10-05T19:36:42.918 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:43.093 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.69-py3-none-any.whl (132 kB) 2023-10-05T19:36:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:43.301 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.68-py3-none-any.whl (132 kB) 2023-10-05T19:36:43.335 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:43.544 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.67-py3-none-any.whl (132 kB) 2023-10-05T19:36:43.577 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:43.795 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.66-py3-none-any.whl (132 kB) 2023-10-05T19:36:43.827 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.65-py3-none-any.whl (132 kB) 2023-10-05T19:36:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.5 MB/s eta 0:00:00 2023-10-05T19:36:44.227 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.64-py3-none-any.whl (132 kB) 2023-10-05T19:36:44.270 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.1 MB/s eta 0:00:00 2023-10-05T19:36:44.447 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.63-py3-none-any.whl (132 kB) 2023-10-05T19:36:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:44.659 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.62-py3-none-any.whl (132 kB) 2023-10-05T19:36:44.697 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:44.877 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.61-py3-none-any.whl (132 kB) 2023-10-05T19:36:44.923 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:45.103 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.60-py3-none-any.whl (132 kB) 2023-10-05T19:36:45.140 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:45.323 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.59-py3-none-any.whl (132 kB) 2023-10-05T19:36:45.374 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.6 MB/s eta 0:00:00 2023-10-05T19:36:45.553 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.58-py3-none-any.whl (132 kB) 2023-10-05T19:36:45.591 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:45.770 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.57-py3-none-any.whl (132 kB) 2023-10-05T19:36:45.816 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:46.003 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.56-py3-none-any.whl (132 kB) 2023-10-05T19:36:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.7 MB/s eta 0:00:00 2023-10-05T19:36:46.223 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.55-py3-none-any.whl (132 kB) 2023-10-05T19:36:46.259 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:46.440 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.54-py3-none-any.whl (132 kB) 2023-10-05T19:36:46.476 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.53-py3-none-any.whl (132 kB) 2023-10-05T19:36:46.702 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:46.880 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.52-py3-none-any.whl (132 kB) 2023-10-05T19:36:46.941 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-10-05T19:36:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.51-py3-none-any.whl (132 kB) 2023-10-05T19:36:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:47.417 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.50-py3-none-any.whl (132 kB) 2023-10-05T19:36:47.453 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:47.633 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.49-py3-none-any.whl (132 kB) 2023-10-05T19:36:47.662 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.8 MB/s eta 0:00:00 2023-10-05T19:36:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.48-py3-none-any.whl (132 kB) 2023-10-05T19:36:47.888 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:48.083 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.47-py3-none-any.whl (132 kB) 2023-10-05T19:36:48.122 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.46-py3-none-any.whl (132 kB) 2023-10-05T19:36:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.0 MB/s eta 0:00:00 2023-10-05T19:36:48.563 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.45-py3-none-any.whl (132 kB) 2023-10-05T19:36:48.606 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.1 MB/s eta 0:00:00 2023-10-05T19:36:48.786 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.44-py3-none-any.whl (132 kB) 2023-10-05T19:36:48.824 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:49.041 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.43-py3-none-any.whl (132 kB) 2023-10-05T19:36:49.074 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:49.256 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.42-py3-none-any.whl (132 kB) 2023-10-05T19:36:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:49.482 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.41-py3-none-any.whl (132 kB) 2023-10-05T19:36:49.550 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 1.9 MB/s eta 0:00:00 2023-10-05T19:36:49.732 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.40-py3-none-any.whl (132 kB) 2023-10-05T19:36:49.768 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:49.955 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.39-py3-none-any.whl (132 kB) 2023-10-05T19:36:49.993 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.38-py3-none-any.whl (132 kB) 2023-10-05T19:36:50.243 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:36:50.424 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.37-py3-none-any.whl (132 kB) 2023-10-05T19:36:50.461 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.36-py3-none-any.whl (132 kB) 2023-10-05T19:36:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.4 MB/s eta 0:00:00 2023-10-05T19:36:50.898 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.35-py3-none-any.whl (132 kB) 2023-10-05T19:36:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 2.9 MB/s eta 0:00:00 2023-10-05T19:36:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.34-py3-none-any.whl (132 kB) 2023-10-05T19:36:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.1 MB/s eta 0:00:00 2023-10-05T19:36:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.33-py3-none-any.whl (132 kB) 2023-10-05T19:36:51.421 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 4.3 MB/s eta 0:00:00 2023-10-05T19:36:51.603 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.32-py3-none-any.whl (132 kB) 2023-10-05T19:36:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.7/132.7 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.31-py3-none-any.whl (132 kB) 2023-10-05T19:36:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 2.1 MB/s eta 0:00:00 2023-10-05T19:36:52.072 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.30-py3-none-any.whl (132 kB) 2023-10-05T19:36:52.113 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:52.393 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.29-py3-none-any.whl (132 kB) 2023-10-05T19:36:52.566 INFO:teuthology.orchestra.run.smithi064.stdout: ------------------------------------- 132.6/132.6 kB 728.9 kB/s eta 0:00:00 2023-10-05T19:36:52.757 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.28-py3-none-any.whl (132 kB) 2023-10-05T19:36:52.816 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:36:52.996 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.27-py3-none-any.whl (132 kB) 2023-10-05T19:36:53.033 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.8 MB/s eta 0:00:00 2023-10-05T19:36:53.215 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.26-py3-none-any.whl (132 kB) 2023-10-05T19:36:53.392 INFO:teuthology.orchestra.run.smithi064.stdout: ------------------------------------- 132.6/132.6 kB 711.7 kB/s eta 0:00:00 2023-10-05T19:36:53.577 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.25-py3-none-any.whl (132 kB) 2023-10-05T19:36:53.618 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:36:53.803 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.24-py3-none-any.whl (132 kB) 2023-10-05T19:36:53.918 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 1.1 MB/s eta 0:00:00 2023-10-05T19:36:54.139 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.23-py3-none-any.whl (132 kB) 2023-10-05T19:36:54.167 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 5.0 MB/s eta 0:00:00 2023-10-05T19:36:54.350 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.22-py3-none-any.whl (132 kB) 2023-10-05T19:36:54.403 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 2.6 MB/s eta 0:00:00 2023-10-05T19:36:54.584 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.21-py3-none-any.whl (132 kB) 2023-10-05T19:36:54.628 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:54.809 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.20-py3-none-any.whl (132 kB) 2023-10-05T19:36:54.887 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 1.7 MB/s eta 0:00:00 2023-10-05T19:36:55.065 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.19-py3-none-any.whl (132 kB) 2023-10-05T19:36:55.104 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:55.329 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.18-py3-none-any.whl (132 kB) 2023-10-05T19:36:55.363 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 4.2 MB/s eta 0:00:00 2023-10-05T19:36:55.544 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.17-py3-none-any.whl (132 kB) 2023-10-05T19:36:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.6/132.6 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:55.762 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.16-py3-none-any.whl (132 kB) 2023-10-05T19:36:55.797 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 4.0 MB/s eta 0:00:00 2023-10-05T19:36:55.983 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.15-py3-none-any.whl (132 kB) 2023-10-05T19:36:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 2.0 MB/s eta 0:00:00 2023-10-05T19:36:56.234 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.14-py3-none-any.whl (132 kB) 2023-10-05T19:36:56.273 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.5 MB/s eta 0:00:00 2023-10-05T19:36:56.493 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.13-py3-none-any.whl (132 kB) 2023-10-05T19:36:56.530 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.12-py3-none-any.whl (132 kB) 2023-10-05T19:36:56.750 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:56.931 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.11-py3-none-any.whl (132 kB) 2023-10-05T19:36:56.966 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 4.0 MB/s eta 0:00:00 2023-10-05T19:36:57.148 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.10-py3-none-any.whl (132 kB) 2023-10-05T19:36:57.184 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.9 MB/s eta 0:00:00 2023-10-05T19:36:57.374 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.9-py3-none-any.whl (132 kB) 2023-10-05T19:36:57.409 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 4.0 MB/s eta 0:00:00 2023-10-05T19:36:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.8-py3-none-any.whl (132 kB) 2023-10-05T19:36:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.2 MB/s eta 0:00:00 2023-10-05T19:36:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.7-py3-none-any.whl (132 kB) 2023-10-05T19:36:57.928 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 1.6 MB/s eta 0:00:00 2023-10-05T19:36:58.115 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.6-py3-none-any.whl (132 kB) 2023-10-05T19:36:58.152 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:36:58.337 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.5-py3-none-any.whl (132 kB) 2023-10-05T19:36:58.387 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 2.7 MB/s eta 0:00:00 2023-10-05T19:36:58.571 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.4-py3-none-any.whl (132 kB) 2023-10-05T19:36:58.612 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:58.796 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.3-py3-none-any.whl (132 kB) 2023-10-05T19:36:58.836 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:36:59.019 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.2-py3-none-any.whl (132 kB) 2023-10-05T19:36:59.063 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.1 MB/s eta 0:00:00 2023-10-05T19:36:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.1-py3-none-any.whl (132 kB) 2023-10-05T19:36:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.7 MB/s eta 0:00:00 2023-10-05T19:36:59.466 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.26.0-py3-none-any.whl (132 kB) 2023-10-05T19:36:59.520 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 2.5 MB/s eta 0:00:00 2023-10-05T19:36:59.739 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.5-py3-none-any.whl (132 kB) 2023-10-05T19:36:59.809 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 1.9 MB/s eta 0:00:00 2023-10-05T19:36:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.4-py3-none-any.whl (132 kB) 2023-10-05T19:36:59.990 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:37:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.3-py3-none-any.whl (132 kB) 2023-10-05T19:37:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:37:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.2-py3-none-any.whl (132 kB) 2023-10-05T19:37:00.341 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.3 MB/s eta 0:00:00 2023-10-05T19:37:00.477 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.1-py3-none-any.whl (132 kB) 2023-10-05T19:37:00.517 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.4 MB/s eta 0:00:00 2023-10-05T19:37:00.654 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.25.0-py3-none-any.whl (132 kB) 2023-10-05T19:37:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 3.6 MB/s eta 0:00:00 2023-10-05T19:37:00.913 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading boto3-1.24.96-py3-none-any.whl (132 kB) 2023-10-05T19:37:00.972 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 132.5/132.5 kB 2.3 MB/s eta 0:00:00 2023-10-05T19:37:01.150 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3>=1.0.0->-r requirements.txt (line 4)) 2023-10-05T19:37:01.170 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-10-05T19:37:01.265 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting s3transfer<0.7.0,>=0.6.0 (from boto3>=1.0.0->-r requirements.txt (line 4)) 2023-10-05T19:37:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for s3transfer<0.7.0,>=0.6.0 from https://files.pythonhosted.org/packages/d9/17/a3b666f5ef9543cfd3c661d39d1e193abb9649d0cfbbfee3cf3b51d5af02/s3transfer-0.6.2-py3-none-any.whl.metadata 2023-10-05T19:37:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading s3transfer-0.6.2-py3-none-any.whl.metadata (1.8 kB) 2023-10-05T19:37:01.391 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.28.0->-r requirements.txt (line 6)) 2023-10-05T19:37:01.409 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-10-05T19:37:01.445 INFO:teuthology.orchestra.run.smithi064.stdout: --------------------------------------- 247.7/247.7 kB 7.4 MB/s eta 0:00:00 2023-10-05T19:37:01.604 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.28.0->-r requirements.txt (line 6)) 2023-10-05T19:37:01.605 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for urllib3<1.27,>=1.25.4 from https://files.pythonhosted.org/packages/48/fe/a5c6cc46e9fe9171d7ecf0f33ee7aae14642f8d74baa7af4d7840f9358be/urllib3-1.26.17-py2.py3-none-any.whl.metadata 2023-10-05T19:37:01.622 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading urllib3-1.26.17-py2.py3-none-any.whl.metadata (48 kB) 2023-10-05T19:37:01.663 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 48.7/48.7 kB 1.0 MB/s eta 0:00:00 2023-10-05T19:37:01.767 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting zope.event (from gevent>=1.0->-r requirements.txt (line 9)) 2023-10-05T19:37:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for zope.event from https://files.pythonhosted.org/packages/fe/42/f8dbc2b9ad59e927940325a22d6d3931d630c3644dae7e2369ef5d9ba230/zope.event-5.0-py3-none-any.whl.metadata 2023-10-05T19:37:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading zope.event-5.0-py3-none-any.whl.metadata (4.4 kB) 2023-10-05T19:37:02.059 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting zope.interface (from gevent>=1.0->-r requirements.txt (line 9)) 2023-10-05T19:37:02.060 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for zope.interface from https://files.pythonhosted.org/packages/46/2e/777a4c4a95e44c5d6200c089369bfb59bf14b0bc22afd6e327b4c7878515/zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:37:02.078 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (41 kB) 2023-10-05T19:37:02.106 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 41.7/41.7 kB 1.3 MB/s eta 0:00:00 2023-10-05T19:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting greenlet>=2.0.0 (from gevent>=1.0->-r requirements.txt (line 9)) 2023-10-05T19:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for greenlet>=2.0.0 from https://files.pythonhosted.org/packages/75/5a/4ecc17749c54882dd3422befcd35fec5128ae2a72138844899322af0b55a/greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata 2023-10-05T19:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.8 kB) 2023-10-05T19:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting six (from isodate>=0.4.4->-r requirements.txt (line 10)) 2023-10-05T19:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T19:37:02.629 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting charset-normalizer<4,>=2 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-10-05T19:37:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/1e/c8/fd52271326c052f95f47ef718b018aa2bc3fd097d9bac44d7d48894c6130/charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata 2023-10-05T19:37:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (32 kB) 2023-10-05T19:37:02.700 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting idna<4,>=2.5 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-10-05T19:37:02.701 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T19:37:02.793 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting certifi>=2017.4.17 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-10-05T19:37:02.793 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata 2023-10-05T19:37:02.794 INFO:teuthology.orchestra.run.smithi064.stdout: Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB) 2023-10-05T19:37:02.952 INFO:teuthology.orchestra.run.smithi064.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 (from httplib2->-r requirements.txt (line 13)) 2023-10-05T19:37:02.952 INFO:teuthology.orchestra.run.smithi064.stdout: Obtaining dependency information for pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 from https://files.pythonhosted.org/packages/39/92/8486ede85fcc088f1b3dba4ce92dd29d126fd96b0008ea213167940a2475/pyparsing-3.1.1-py3-none-any.whl.metadata 2023-10-05T19:37:02.970 INFO:teuthology.orchestra.run.smithi064.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB) 2023-10-05T19:37:03.282 INFO:teuthology.orchestra.run.smithi064.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.8/site-packages (from zope.event->gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-10-05T19:37:03.328 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB) 2023-10-05T19:37:03.375 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 736.6/736.6 kB 16.3 MB/s eta 0:00:00 2023-10-05T19:37:03.413 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-10-05T19:37:03.464 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading gevent-23.9.1-cp38-cp38-manylinux_2_28_x86_64.whl (6.5 MB) 2023-10-05T19:37:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 6.5/6.5 MB 36.2 MB/s eta 0:00:00 2023-10-05T19:37:03.765 INFO:teuthology.orchestra.run.smithi064.stdout:Using cached requests-2.31.0-py3-none-any.whl (62 kB) 2023-10-05T19:37:03.765 INFO:teuthology.orchestra.run.smithi064.stdout:Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T19:37:03.766 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl (8.0 MB) 2023-10-05T19:37:03.830 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 8.0/8.0 MB 53.7 MB/s eta 0:00:00 2023-10-05T19:37:03.833 INFO:teuthology.orchestra.run.smithi064.stdout:Using cached certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T19:37:03.837 INFO:teuthology.orchestra.run.smithi064.stdout:Using cached charset_normalizer-3.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (137 kB) 2023-10-05T19:37:03.840 INFO:teuthology.orchestra.run.smithi064.stdout:Using cached greenlet-3.0.0-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (617 kB) 2023-10-05T19:37:03.860 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-10-05T19:37:03.921 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 103.1/103.1 kB 1.7 MB/s eta 0:00:00 2023-10-05T19:37:03.941 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading s3transfer-0.6.2-py3-none-any.whl (79 kB) 2023-10-05T19:37:03.972 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 79.8/79.8 kB 2.7 MB/s eta 0:00:00 2023-10-05T19:37:03.993 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading urllib3-1.26.17-py2.py3-none-any.whl (143 kB) 2023-10-05T19:37:04.064 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 143.4/143.4 kB 2.0 MB/s eta 0:00:00 2023-10-05T19:37:04.085 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading zope.event-5.0-py3-none-any.whl (6.8 kB) 2023-10-05T19:37:04.132 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (249 kB) 2023-10-05T19:37:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: ---------------------------------------- 249.3/249.3 kB 4.1 MB/s eta 0:00:00 2023-10-05T19:37:04.391 INFO:teuthology.orchestra.run.smithi064.stdout:Installing collected packages: pytz, nose, boto, zope.interface, zope.event, urllib3, six, PyYAML, pyparsing, munch, lxml, jmespath, idna, greenlet, charset-normalizer, certifi, requests, python-dateutil, isodate, httplib2, gevent, botocore, s3transfer, boto3 2023-10-05T19:37:08.408 INFO:teuthology.orchestra.run.smithi064.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.24.96 botocore-1.27.96 certifi-2023.7.22 charset-normalizer-3.3.0 gevent-23.9.1 greenlet-3.0.0 httplib2-0.22.0 idna-3.4 isodate-0.6.1 jmespath-1.0.1 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.31.0 s3transfer-0.6.2 six-1.16.0 urllib3-1.26.17 zope.event-5.0 zope.interface-6.1 2023-10-05T19:37:08.845 INFO:teuthology.orchestra.run.smithi064.stdout:running develop 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:running egg_info 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:creating s3tests.egg-info 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:writing s3tests.egg-info/PKG-INFO 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-10-05T19:37:08.866 INFO:teuthology.orchestra.run.smithi064.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-10-05T19:37:08.867 INFO:teuthology.orchestra.run.smithi064.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:37:08.897 INFO:teuthology.orchestra.run.smithi064.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:37:08.898 INFO:teuthology.orchestra.run.smithi064.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:37:08.899 INFO:teuthology.orchestra.run.smithi064.stdout:running build_ext 2023-10-05T19:37:08.899 INFO:teuthology.orchestra.run.smithi064.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages/s3tests.egg-link (link to .) 2023-10-05T19:37:08.899 INFO:teuthology.orchestra.run.smithi064.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-10-05T19:37:08.900 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.900 INFO:teuthology.orchestra.run.smithi064.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:37:08.901 INFO:teuthology.orchestra.run.smithi064.stdout:Processing dependencies for s3tests==0.0.1 2023-10-05T19:37:08.969 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for isodate==0.6.1 2023-10-05T19:37:08.969 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: isodate 0.6.1 2023-10-05T19:37:08.969 INFO:teuthology.orchestra.run.smithi064.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-10-05T19:37:08.969 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.969 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for gevent==23.9.1 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: gevent 23.9.1 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout:Adding gevent 23.9.1 to easy-install.pth file 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.970 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for munch==4.0.0 2023-10-05T19:37:08.971 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: munch 4.0.0 2023-10-05T19:37:08.971 INFO:teuthology.orchestra.run.smithi064.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for PyYAML==6.0.1 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: PyYAML 6.0.1 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for boto3==1.24.96 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: boto3 1.24.96 2023-10-05T19:37:08.972 INFO:teuthology.orchestra.run.smithi064.stdout:Adding boto3 1.24.96 to easy-install.pth file 2023-10-05T19:37:08.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for boto==2.49.0 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: boto 2.49.0 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for six==1.16.0 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: six 1.16.0 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout:Adding six 1.16.0 to easy-install.pth file 2023-10-05T19:37:08.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.975 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for greenlet==3.0.0 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: greenlet 3.0.0 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Adding greenlet 3.0.0 to easy-install.pth file 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for zope.event==5.0 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: zope.event 5.0 2023-10-05T19:37:08.976 INFO:teuthology.orchestra.run.smithi064.stdout:Adding zope.event 5.0 to easy-install.pth file 2023-10-05T19:37:08.977 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.977 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.977 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for zope.interface==6.1 2023-10-05T19:37:08.977 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: zope.interface 6.1 2023-10-05T19:37:08.977 INFO:teuthology.orchestra.run.smithi064.stdout:Adding zope.interface 6.1 to easy-install.pth file 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for jmespath==1.0.1 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: jmespath 1.0.1 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout:Adding jmespath 1.0.1 to easy-install.pth file 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.978 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.979 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for botocore==1.27.96 2023-10-05T19:37:08.979 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: botocore 1.27.96 2023-10-05T19:37:08.979 INFO:teuthology.orchestra.run.smithi064.stdout:Adding botocore 1.27.96 to easy-install.pth file 2023-10-05T19:37:08.979 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.979 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.980 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for s3transfer==0.6.2 2023-10-05T19:37:08.980 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: s3transfer 0.6.2 2023-10-05T19:37:08.980 INFO:teuthology.orchestra.run.smithi064.stdout:Adding s3transfer 0.6.2 to easy-install.pth file 2023-10-05T19:37:08.980 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for setuptools==44.1.0 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: setuptools 44.1.0 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.982 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for urllib3==1.26.17 2023-10-05T19:37:08.982 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: urllib3 1.26.17 2023-10-05T19:37:08.982 INFO:teuthology.orchestra.run.smithi064.stdout:Adding urllib3 1.26.17 to easy-install.pth file 2023-10-05T19:37:08.982 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.982 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout:Searching for python-dateutil==2.8.2 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout:Best match: python-dateutil 2.8.2 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.8/site-packages 2023-10-05T19:37:08.983 INFO:teuthology.orchestra.run.smithi064.stdout:Finished processing dependencies for s3tests==0.0.1 2023-10-05T19:37:09.002 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:37:09.003 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-10-05T19:37:09.009 INFO:tasks.s3tests:Configuring boto... 2023-10-05T19:37:09.010 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-05T19:37:09.010 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T19:37:09.062 DEBUG:teuthology.orchestra.run.smithi064:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr:/home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_headers.py:126: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr: if boto_type is 'S3Connection': 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr:/home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_headers.py:131: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr: elif boto_type is 'HTTPRequest': 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr:/home/ubuntu/cephtest/s3-tests-client.0/s3tests/functional/test_s3_website.py:192: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-10-05T19:37:09.516 INFO:teuthology.orchestra.run.smithi064.stderr: if k is 'BucketName': 2023-10-05T19:37:09.537 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-10-05T19:37:09.555 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-10-05T19:37:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-10-05T19:37:09.649 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-10-05T19:37:09.671 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-10-05T19:37:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-10-05T19:37:09.721 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-10-05T19:37:09.742 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-10-05T19:37:09.789 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-10-05T19:37:09.791 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-10-05T19:37:09.815 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-10-05T19:37:09.837 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-10-05T19:37:09.855 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:37:09.873 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-10-05T19:37:09.876 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:37:09.878 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-10-05T19:37:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-10-05T19:37:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-10-05T19:37:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-10-05T19:37:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:37:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:37:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:37:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:37:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:37:09.881 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:37:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:37:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:37:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:37:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:37:09.883 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:37:09.884 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:37:09.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:37:09.942 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-10-05T19:37:09.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-10-05T19:37:09.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-10-05T19:37:09.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-10-05T19:37:09.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-10-05T19:37:09.944 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-10-05T19:37:09.944 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-10-05T19:37:10.049 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-10-05T19:37:10.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-10-05T19:37:11.865 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-10-05T19:37:11.925 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-10-05T19:37:11.973 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-10-05T19:37:12.000 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-10-05T19:37:12.129 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-10-05T19:37:12.129 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-10-05T19:37:12.129 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-10-05T19:37:12.130 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:37:12.131 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:37:12.133 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-10-05T19:37:12.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-10-05T19:37:12.135 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-10-05T19:37:12.135 INFO:teuthology.orchestra.run.smithi064.stderr:SKIP 2023-10-05T19:37:14.333 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-10-05T19:37:14.365 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-10-05T19:37:25.056 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-10-05T19:37:25.092 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-10-05T19:37:25.371 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-10-05T19:37:25.371 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-10-05T19:37:25.371 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-10-05T19:37:25.371 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-10-05T19:37:25.389 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-10-05T19:37:25.444 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-10-05T19:37:25.486 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-10-05T19:37:25.552 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-10-05T19:37:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-10-05T19:37:25.611 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-10-05T19:37:25.638 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-10-05T19:37:25.648 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-10-05T19:37:25.683 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-10-05T19:37:25.741 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:37:25.798 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-10-05T19:37:25.837 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:37:25.874 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-10-05T19:37:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:37:25.941 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:37:25.974 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-10-05T19:37:25.998 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:37:26.024 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-10-05T19:37:26.036 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:37:26.045 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-10-05T19:37:26.053 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:37:26.061 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-10-05T19:37:26.070 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:37:26.121 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-10-05T19:37:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-10-05T19:37:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-10-05T19:37:26.358 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-10-05T19:37:26.493 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-10-05T19:37:26.591 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-10-05T19:37:26.681 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-10-05T19:37:26.774 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-10-05T19:37:26.903 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-10-05T19:37:27.093 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-10-05T19:37:27.282 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-10-05T19:37:27.336 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:37:27.384 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:37:27.471 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-10-05T19:37:27.559 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-10-05T19:37:27.741 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-10-05T19:37:27.955 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-10-05T19:37:28.040 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-10-05T19:37:28.125 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-10-05T19:37:28.217 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-10-05T19:37:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-10-05T19:37:28.397 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-10-05T19:37:28.490 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-10-05T19:37:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-10-05T19:37:28.670 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-10-05T19:37:28.756 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-10-05T19:37:28.858 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-10-05T19:37:28.953 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-10-05T19:37:29.046 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-10-05T19:37:29.125 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-10-05T19:37:29.232 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-10-05T19:37:29.337 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-10-05T19:37:29.433 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-10-05T19:37:29.521 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-10-05T19:37:41.940 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-10-05T19:37:42.020 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-10-05T19:37:42.096 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-10-05T19:37:42.199 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-10-05T19:37:42.269 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-10-05T19:37:42.345 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-10-05T19:37:42.427 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-10-05T19:37:42.513 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-10-05T19:37:42.591 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-10-05T19:37:42.666 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-10-05T19:37:42.742 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-10-05T19:37:42.827 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-10-05T19:37:42.913 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-10-05T19:37:43.013 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-10-05T19:37:43.105 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-10-05T19:37:43.191 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-10-05T19:37:43.284 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-10-05T19:37:43.372 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:37:43.458 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:37:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:37:43.675 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:37:43.760 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:37:43.844 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:37:43.947 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-10-05T19:37:44.042 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-10-05T19:37:44.124 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-10-05T19:37:44.214 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-10-05T19:37:44.310 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-10-05T19:37:44.400 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-10-05T19:37:44.810 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-10-05T19:37:44.857 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-10-05T19:37:45.308 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-10-05T19:37:45.777 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-10-05T19:37:45.871 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-10-05T19:37:45.961 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-10-05T19:37:46.050 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-10-05T19:37:46.138 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-10-05T19:37:46.240 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-10-05T19:37:46.331 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-10-05T19:37:46.412 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-10-05T19:37:46.499 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-10-05T19:37:46.591 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-10-05T19:37:46.690 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-10-05T19:37:46.779 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-10-05T19:37:46.877 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-10-05T19:37:47.079 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-10-05T19:37:47.305 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-10-05T19:37:47.358 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-10-05T19:37:47.409 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-10-05T19:37:47.450 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-10-05T19:37:47.493 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-10-05T19:37:47.501 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-10-05T19:37:47.510 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-10-05T19:37:47.520 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-10-05T19:37:47.583 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-10-05T19:37:47.817 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-10-05T19:37:47.828 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-10-05T19:37:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-10-05T19:37:47.963 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-10-05T19:37:48.005 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-10-05T19:37:48.154 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-10-05T19:37:48.257 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-10-05T19:38:00.777 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-10-05T19:38:13.418 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-10-05T19:38:13.466 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-10-05T19:38:13.507 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-10-05T19:38:13.558 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-10-05T19:38:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-10-05T19:38:13.691 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-10-05T19:38:13.745 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-10-05T19:38:13.797 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-10-05T19:38:13.869 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-10-05T19:38:13.934 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-10-05T19:38:13.988 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-10-05T19:38:14.037 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-10-05T19:38:14.087 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-10-05T19:38:14.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-10-05T19:38:14.170 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-10-05T19:38:14.239 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-10-05T19:38:14.274 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-10-05T19:38:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-10-05T19:38:14.436 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-10-05T19:38:14.476 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-10-05T19:38:14.524 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-10-05T19:38:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-10-05T19:38:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-10-05T19:38:14.666 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-10-05T19:38:14.705 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-10-05T19:38:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-10-05T19:38:14.774 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-10-05T19:38:14.812 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-10-05T19:38:14.850 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-10-05T19:38:14.900 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-10-05T19:38:14.938 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-10-05T19:38:14.973 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-10-05T19:38:15.004 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-10-05T19:38:15.040 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-10-05T19:38:15.075 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-10-05T19:38:15.111 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-10-05T19:38:15.147 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-10-05T19:38:15.184 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-10-05T19:38:15.217 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-10-05T19:38:15.254 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-10-05T19:38:15.287 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-10-05T19:38:15.321 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-10-05T19:38:15.374 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-10-05T19:38:15.427 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-10-05T19:38:15.484 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-10-05T19:38:15.533 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-10-05T19:38:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-10-05T19:38:16.644 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-10-05T19:38:16.703 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-10-05T19:38:16.762 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-10-05T19:38:16.828 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-10-05T19:38:16.937 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-10-05T19:38:17.057 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-10-05T19:38:17.111 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-10-05T19:38:17.221 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-10-05T19:38:17.294 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-10-05T19:38:17.353 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-10-05T19:38:17.418 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-10-05T19:38:17.466 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-10-05T19:38:17.604 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-10-05T19:38:17.776 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-10-05T19:38:17.841 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-10-05T19:38:17.879 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-10-05T19:38:17.889 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-10-05T19:38:17.986 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-10-05T19:38:18.037 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-10-05T19:38:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-10-05T19:38:18.142 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-10-05T19:38:18.199 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-10-05T19:38:18.249 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-10-05T19:38:18.298 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-10-05T19:38:18.431 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-10-05T19:38:18.495 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-10-05T19:38:18.547 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-10-05T19:38:18.594 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-10-05T19:38:18.643 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-10-05T19:38:18.697 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-10-05T19:38:18.757 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-10-05T19:38:18.820 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-10-05T19:38:18.864 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-10-05T19:38:18.921 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-10-05T19:38:18.935 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-10-05T19:38:18.948 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-10-05T19:38:18.962 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-10-05T19:38:18.975 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-10-05T19:38:19.012 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-10-05T19:38:19.049 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-10-05T19:38:19.089 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-10-05T19:38:19.128 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-10-05T19:38:19.170 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-10-05T19:38:19.209 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-10-05T19:38:19.249 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-10-05T19:38:19.292 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-10-05T19:38:19.330 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-10-05T19:38:19.363 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-10-05T19:38:19.414 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-10-05T19:38:19.465 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-10-05T19:38:19.533 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-10-05T19:38:19.583 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-10-05T19:38:19.632 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-10-05T19:38:19.771 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-10-05T19:38:19.849 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-10-05T19:38:19.925 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-10-05T19:38:20.006 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-10-05T19:38:20.049 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-10-05T19:38:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-10-05T19:38:20.240 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-10-05T19:38:20.354 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-10-05T19:38:20.474 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-10-05T19:38:20.549 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-10-05T19:38:20.662 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-10-05T19:38:20.756 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-10-05T19:38:20.845 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-10-05T19:38:20.948 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-10-05T19:38:21.075 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-10-05T19:38:21.122 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-10-05T19:38:21.216 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-10-05T19:38:21.277 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-10-05T19:38:21.340 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-10-05T19:38:21.394 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-10-05T19:38:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-10-05T19:38:21.566 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-10-05T19:38:21.699 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-10-05T19:38:21.827 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-10-05T19:38:21.952 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-10-05T19:38:22.073 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-10-05T19:38:22.202 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-10-05T19:38:22.370 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-10-05T19:38:22.494 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-10-05T19:38:22.628 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-10-05T19:38:22.759 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-10-05T19:38:22.989 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-10-05T19:38:23.185 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-10-05T19:38:23.378 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-10-05T19:38:23.633 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-10-05T19:38:23.840 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-10-05T19:38:23.847 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-10-05T19:38:23.858 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-10-05T19:38:23.868 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-10-05T19:38:24.053 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-10-05T19:38:24.241 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-10-05T19:38:24.270 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-10-05T19:38:24.301 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-10-05T19:38:24.436 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-10-05T19:38:25.064 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-10-05T19:38:25.191 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-10-05T19:38:25.292 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-10-05T19:38:25.373 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-10-05T19:38:25.493 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-10-05T19:38:25.579 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-10-05T19:38:25.660 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-10-05T19:38:25.818 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-10-05T19:38:25.895 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-10-05T19:38:26.021 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-10-05T19:38:26.166 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-10-05T19:38:26.347 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-10-05T19:38:26.522 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-10-05T19:38:26.570 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-10-05T19:38:26.618 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-10-05T19:38:27.061 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-10-05T19:38:27.230 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-10-05T19:38:28.561 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-10-05T19:38:28.617 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-10-05T19:38:28.723 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-10-05T19:38:29.026 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-10-05T19:38:29.100 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-10-05T19:38:29.271 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-10-05T19:38:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-10-05T19:38:31.358 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-10-05T19:38:33.185 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-10-05T19:38:39.344 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-10-05T19:38:40.629 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-10-05T19:38:42.108 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-10-05T19:38:42.339 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-10-05T19:38:42.833 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-10-05T19:38:43.272 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-10-05T19:38:43.533 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-10-05T19:38:43.594 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-10-05T19:38:44.033 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-10-05T19:38:44.141 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-10-05T19:38:44.274 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-10-05T19:38:44.403 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-10-05T19:38:44.453 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-10-05T19:38:44.525 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-10-05T19:38:47.692 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-10-05T19:38:50.753 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-10-05T19:38:53.805 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-10-05T19:38:53.877 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-10-05T19:38:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-10-05T19:38:54.314 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-10-05T19:38:54.782 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-10-05T19:38:55.043 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-10-05T19:38:55.430 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-10-05T19:38:56.072 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-10-05T19:38:56.290 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-10-05T19:38:56.688 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-10-05T19:38:57.342 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-10-05T19:38:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-10-05T19:38:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-10-05T19:38:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-10-05T19:38:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-10-05T19:39:01.990 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-10-05T19:39:02.040 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-10-05T19:39:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-10-05T19:39:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-10-05T19:39:02.202 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-10-05T19:39:02.306 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-10-05T19:39:06.591 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-10-05T19:39:07.068 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-10-05T19:39:07.245 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-10-05T19:39:07.482 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-10-05T19:39:07.682 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-10-05T19:39:09.239 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-10-05T19:39:11.070 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-10-05T19:39:18.427 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-10-05T19:39:20.401 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-10-05T19:39:20.938 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-10-05T19:39:21.469 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-10-05T19:39:21.713 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-10-05T19:39:22.050 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-10-05T19:39:22.134 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-10-05T19:39:22.423 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-10-05T19:39:22.598 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-10-05T19:39:23.169 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-10-05T19:39:23.219 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-10-05T19:39:23.281 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-10-05T19:39:23.348 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-10-05T19:39:23.401 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-10-05T19:39:23.443 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-10-05T19:39:23.492 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-10-05T19:39:23.544 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-10-05T19:39:23.586 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-10-05T19:39:23.667 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-10-05T19:39:23.722 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-10-05T19:39:23.782 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-10-05T19:39:23.836 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-10-05T19:39:23.897 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-10-05T19:39:23.942 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-10-05T19:39:23.942 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-10-05T19:39:23.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-10-05T19:39:23.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-10-05T19:39:23.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-10-05T19:39:23.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-10-05T19:39:23.943 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-10-05T19:39:24.036 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-10-05T19:39:24.093 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-10-05T19:39:24.174 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-10-05T19:39:24.231 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-10-05T19:39:24.300 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-10-05T19:39:24.360 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-10-05T19:39:24.424 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-10-05T19:39:24.481 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-10-05T19:39:24.538 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-10-05T19:39:24.582 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-10-05T19:39:24.634 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-10-05T19:39:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-10-05T19:39:26.783 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-10-05T19:39:26.840 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-10-05T19:39:26.969 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-10-05T19:39:27.053 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-10-05T19:39:27.099 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-10-05T19:39:27.138 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-10-05T19:39:29.024 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-10-05T19:39:29.834 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-10-05T19:39:30.679 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-10-05T19:39:30.758 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-10-05T19:39:30.839 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-10-05T19:39:30.932 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-10-05T19:39:31.041 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-10-05T19:39:31.124 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-10-05T19:39:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-10-05T19:39:31.265 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-10-05T19:39:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-10-05T19:39:31.434 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-10-05T19:39:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-10-05T19:39:31.771 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-10-05T19:39:31.966 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-10-05T19:39:32.166 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-10-05T19:39:32.356 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-10-05T19:39:32.536 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-10-05T19:39:32.709 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-10-05T19:39:32.897 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-10-05T19:39:33.090 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-10-05T19:39:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-10-05T19:39:33.518 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-10-05T19:39:33.770 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-10-05T19:39:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-10-05T19:39:33.894 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-10-05T19:39:33.955 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-10-05T19:39:34.162 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-10-05T19:39:34.373 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-10-05T19:39:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-10-05T19:39:34.811 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-10-05T19:39:37.009 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-10-05T19:39:37.155 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-10-05T19:39:37.317 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-10-05T19:39:37.475 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-10-05T19:39:37.713 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-10-05T19:39:37.875 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-10-05T19:39:37.951 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-10-05T19:39:38.137 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-10-05T19:39:38.193 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-10-05T19:39:38.242 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-10-05T19:39:38.290 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-10-05T19:39:38.340 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-10-05T19:39:38.429 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-10-05T19:39:38.481 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-10-05T19:39:38.663 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-10-05T19:39:38.763 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-10-05T19:39:38.805 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-10-05T19:39:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-10-05T19:39:38.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-10-05T19:39:38.923 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-10-05T19:39:38.966 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-10-05T19:39:39.003 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-10-05T19:39:39.036 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-10-05T19:39:39.080 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-10-05T19:39:39.124 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-10-05T19:39:39.230 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-10-05T19:39:39.332 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-10-05T19:39:39.454 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-10-05T19:39:39.614 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-10-05T19:39:39.734 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-10-05T19:39:39.794 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-10-05T19:39:39.967 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-10-05T19:39:40.130 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-10-05T19:39:40.296 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-10-05T19:39:40.447 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-10-05T19:39:40.614 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-10-05T19:39:40.773 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-10-05T19:39:40.971 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-10-05T19:39:41.157 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-10-05T19:39:41.254 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-10-05T19:39:41.362 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-10-05T19:39:41.438 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-10-05T19:39:41.593 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-10-05T19:39:41.651 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-10-05T19:39:41.786 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-10-05T19:39:41.885 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-10-05T19:39:42.020 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-10-05T19:39:42.139 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-10-05T19:39:42.217 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-10-05T19:39:42.294 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-10-05T19:39:42.368 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-10-05T19:39:42.442 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-10-05T19:39:42.517 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-10-05T19:39:42.565 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-10-05T19:39:42.633 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-10-05T19:39:42.695 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-10-05T19:39:42.758 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-10-05T19:39:42.813 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-10-05T19:39:42.900 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-10-05T19:39:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-10-05T19:39:42.995 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-10-05T19:39:43.072 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-10-05T19:39:43.141 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-10-05T19:39:43.206 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-10-05T19:39:43.325 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-10-05T19:39:44.475 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-10-05T19:39:44.535 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-10-05T19:39:44.596 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-10-05T19:39:44.710 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-10-05T19:39:44.813 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-10-05T19:39:45.934 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-10-05T19:39:47.047 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-10-05T19:39:47.112 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-10-05T19:39:47.636 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-10-05T19:39:48.366 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-10-05T19:39:48.648 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-10-05T19:39:48.930 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-10-05T19:39:49.003 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-10-05T19:39:49.715 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-10-05T19:39:50.174 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-10-05T19:39:50.870 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-10-05T19:39:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-10-05T19:39:51.593 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-10-05T19:39:51.826 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-10-05T19:39:51.998 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-10-05T19:39:52.249 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-10-05T19:39:52.564 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-10-05T19:39:52.693 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-10-05T19:39:52.914 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-10-05T19:39:53.160 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-10-05T19:39:53.414 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-10-05T19:39:53.674 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-10-05T19:39:53.952 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-10-05T19:39:53.952 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-10-05T19:39:54.401 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-10-05T19:39:54.850 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-10-05T19:39:55.126 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-10-05T19:39:55.336 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-10-05T19:39:55.527 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-10-05T19:40:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-10-05T19:40:06.006 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-10-05T19:41:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-10-05T19:41:15.582 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:41:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:---------------------------------------------------------------------- 2023-10-05T19:41:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:Ran 618 tests in 246.283s 2023-10-05T19:41:15.582 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:41:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:OK (SKIP=79) 2023-10-05T19:41:15.609 INFO:teuthology.orchestra.run.smithi064.stderr:Exception ignored in: .generate at 0x7fa44a241cf0> 2023-10-05T19:41:15.609 INFO:teuthology.orchestra.run.smithi064.stderr:RuntimeError: generator ignored GeneratorExit 2023-10-05T19:41:15.690 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-10-05T19:41:15.691 DEBUG:teuthology.orchestra.run.smithi064:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-10-05T19:41:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:41:15.755 INFO:tasks.s3tests:Cleaning up boto... 2023-10-05T19:41:15.755 DEBUG:teuthology.orchestra.run.smithi064:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T19:41:15.761 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-10-05T19:41:15.799 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:41:15.799 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:41:15.812 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.807+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 realm 2023-10-05T19:41:15.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:15.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:15.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.811+0000 7fb565429e40 20 searching for the correct realm 2023-10-05T19:41:15.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:15.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:15.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:15.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:15.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:15.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.831+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:15.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.835+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:15.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:15.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:15.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:15.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:15.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:15.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:15.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:15.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:15.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.851+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 zone default found 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 4 Realm: () 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 zonegroup default 2023-10-05T19:41:15.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:15.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 started sync module instance, tier type = 2023-10-05T19:41:15.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.855+0000 7fb565429e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=0 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=1 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=2 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=3 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=4 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=5 2023-10-05T19:41:15.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=6 2023-10-05T19:41:15.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 20 add_watcher() i=7 2023-10-05T19:41:15.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.883+0000 7fb565429e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:41:15.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.887+0000 7fb4affff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:15.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.887+0000 7fb565429e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:41:15.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.887+0000 7fb565429e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:41:15.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb49effd700 20 reqs_thread_entry: start 2023-10-05T19:41:15.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb49cff9700 20 reqs_thread_entry: start 2023-10-05T19:41:15.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 20 init_complete bucket index max shards: 11 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb496ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:41:46 2023 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb4967fc700 20 reqs_thread_entry: start 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 20 get_system_obj_state: rctx=0x7ffd9300b808 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:15.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:15.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.983+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T19:41:15.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T19:41:15.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:41:15.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:15.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 20 get_system_obj_state: rctx=0x7ffd9300b4b0 obj=default.rgw.meta:root:.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:15.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.987+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 : miss 2023-10-05T19:41:15.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 info.flags=0x16 2023-10-05T19:41:15.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 adding default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 to cache LRU end 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 20 Read xattr: user.rgw.acl 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=364 2023-10-05T19:41:15.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 info.flags=0x13 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 moving default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 to cache LRU end 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cls_bucket_list_unordered: got 8718EC 2023-10-05T19:41:15.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cls_bucket_list_unordered: got A27CF1 2023-10-05T19:41:15.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cls_bucket_list_unordered: got 18978B 2023-10-05T19:41:15.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.991+0000 7fb565429e40 10 cls_bucket_list_unordered: got CDCE92 2023-10-05T19:41:15.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 3169C6 2023-10-05T19:41:15.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 6FB8D8 2023-10-05T19:41:15.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 6500F6 2023-10-05T19:41:15.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got E2EC37 2023-10-05T19:41:15.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got F1CE51 2023-10-05T19:41:15.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 24045F 2023-10-05T19:41:16.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 3F0709 2023-10-05T19:41:16.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got 7E05D6 2023-10-05T19:41:16.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.995+0000 7fb565429e40 10 cls_bucket_list_unordered: got A9DC60 2023-10-05T19:41:16.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 627D11 2023-10-05T19:41:16.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got FD1E23 2023-10-05T19:41:16.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 039E88 2023-10-05T19:41:16.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 202858 2023-10-05T19:41:16.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 53C89A 2023-10-05T19:41:16.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 7F64C7 2023-10-05T19:41:16.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 059771 2023-10-05T19:41:16.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got B2F622 2023-10-05T19:41:16.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:15.999+0000 7fb565429e40 10 cls_bucket_list_unordered: got 879558 2023-10-05T19:41:16.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got 1B3918 2023-10-05T19:41:16.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got 3C2970 2023-10-05T19:41:16.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got 425BED 2023-10-05T19:41:16.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got 58ED7D 2023-10-05T19:41:16.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got 59B6CE 2023-10-05T19:41:16.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got EBE004 2023-10-05T19:41:16.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 cls_bucket_list_unordered: got F39F63 2023-10-05T19:41:16.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:8718EC state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 10 manifest: total_size = 360245 2023-10-05T19:41:16.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.11614841613297448763 2023-10-05T19:41:16.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:8718EC state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.003+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.0 2023-10-05T19:41:16.012 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-10-05T19:41:16.012 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.013 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.007+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-10-05T19:41:16.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-10-05T19:41:16.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:A27CF1 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.016 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 10 manifest: total_size = 39911997 2023-10-05T19:41:16.016 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.10492415958923458912 2023-10-05T19:41:16.017 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:A27CF1 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.017 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.017 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.011+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.0 2023-10-05T19:41:16.018 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.015+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=8718EC 2023-10-05T19:41:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.025 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 RGWObjManifest::operator++(): result: ofs=39911997 stripe_ofs=39911997 part_ofs=0 rule->part_size=0 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - tag is: caa95e63-d85f-4735-a9cf-10d220ce9576.4205.10492415958923458912 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_1 2023-10-05T19:41:16.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 333 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_2 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 565 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_3 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 797 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_4 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1029 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_5 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1261 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_6 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1493 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_7 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1725 2023-10-05T19:41:16.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_8 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1957 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - adding obj with name: caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518__shadow_.hmtQWupS4usWIsQ2ah0otMNKPjTbRor_9 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 2189 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.019+0000 7fb565429e40 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: caa95e63-d85f-4735-a9cf-10d220ce9576.4205.10492415958923458912 2023-10-05T19:41:16.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:18978B state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 10 manifest: total_size = 399047 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.8742638050805522222 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:18978B state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.023+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.1 2023-10-05T19:41:16.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.027+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=A27CF1 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:CDCE92 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 10 manifest: total_size = 180000 2023-10-05T19:41:16.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.9110715039263807813 2023-10-05T19:41:16.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:CDCE92 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.031+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.1 2023-10-05T19:41:16.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.035+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=18978B 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3169C6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 10 manifest: total_size = 399065 2023-10-05T19:41:16.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.1495418322909489016 2023-10-05T19:41:16.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3169C6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.039+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.2 2023-10-05T19:41:16.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.043+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=CDCE92 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:6FB8D8 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 10 manifest: total_size = 575736 2023-10-05T19:41:16.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.2119208043054226163 2023-10-05T19:41:16.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:6FB8D8 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.2 2023-10-05T19:41:16.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.047+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=3169C6 2023-10-05T19:41:16.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:6500F6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 10 manifest: total_size = 399052 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.5731327477118970573 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:6500F6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.051+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.3 2023-10-05T19:41:16.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.055+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=6FB8D8 2023-10-05T19:41:16.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:E2EC37 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 10 manifest: total_size = 39 2023-10-05T19:41:16.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.18280689219651541539 2023-10-05T19:41:16.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:E2EC37 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.059+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.3 2023-10-05T19:41:16.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.063+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=6500F6 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:F1CE51 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 10 manifest: total_size = 574678 2023-10-05T19:41:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.17170451613904290432 2023-10-05T19:41:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:F1CE51 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.067+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.3 2023-10-05T19:41:16.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.071+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=E2EC37 2023-10-05T19:41:16.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:24045F state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 10 manifest: total_size = 117 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.12175163873120275994 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:24045F state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.075+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.4 2023-10-05T19:41:16.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.079+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=F1CE51 2023-10-05T19:41:16.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3F0709 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 10 manifest: total_size = 399261 2023-10-05T19:41:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.8269197397205081545 2023-10-05T19:41:16.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3F0709 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.4 2023-10-05T19:41:16.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.083+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=24045F 2023-10-05T19:41:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:7E05D6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 10 manifest: total_size = 5 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.12570627681452280925 2023-10-05T19:41:16.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:7E05D6 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.4 2023-10-05T19:41:16.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.091+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=3F0709 2023-10-05T19:41:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:A9DC60 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 10 manifest: total_size = 399122 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.10478003353232347980 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:A9DC60 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.5 2023-10-05T19:41:16.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.099+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=7E05D6 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:627D11 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 10 manifest: total_size = 399231 2023-10-05T19:41:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.15941647323396477334 2023-10-05T19:41:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:627D11 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.6 2023-10-05T19:41:16.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.107+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=A9DC60 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:FD1E23 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 10 manifest: total_size = 399097 2023-10-05T19:41:16.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.1618567434331967135 2023-10-05T19:41:16.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:FD1E23 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.111+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.6 2023-10-05T19:41:16.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.115+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=627D11 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:039E88 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 10 manifest: total_size = 399109 2023-10-05T19:41:16.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.2533236393650932515 2023-10-05T19:41:16.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:039E88 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.7 2023-10-05T19:41:16.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.119+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=FD1E23 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:202858 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 10 manifest: total_size = 642552 2023-10-05T19:41:16.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.13127841754346773287 2023-10-05T19:41:16.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:202858 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.123+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.7 2023-10-05T19:41:16.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.127+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=039E88 2023-10-05T19:41:16.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:53C89A state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 10 manifest: total_size = 398949 2023-10-05T19:41:16.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.7015265891095314255 2023-10-05T19:41:16.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:53C89A state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.131+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.7 2023-10-05T19:41:16.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.135+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=202858 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:7F64C7 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 10 manifest: total_size = 590086 2023-10-05T19:41:16.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.17352723457221903097 2023-10-05T19:41:16.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:7F64C7 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.139+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.7 2023-10-05T19:41:16.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.143+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=53C89A 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:059771 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 10 manifest: total_size = 5 2023-10-05T19:41:16.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.13722618857187223007 2023-10-05T19:41:16.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:059771 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.147+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.8 2023-10-05T19:41:16.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=7F64C7 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:B2F622 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 10 manifest: total_size = 180000 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.603334769738649885 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:B2F622 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.151+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.8 2023-10-05T19:41:16.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.155+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=059771 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:879558 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 10 manifest: total_size = 399296 2023-10-05T19:41:16.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.18332843291101266405 2023-10-05T19:41:16.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:879558 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.159+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.9 2023-10-05T19:41:16.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=B2F622 2023-10-05T19:41:16.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:1B3918 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 10 manifest: total_size = 398955 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.15922294300570401109 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:1B3918 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.163+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.167+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=879558 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3C2970 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 10 manifest: total_size = 399115 2023-10-05T19:41:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.1761963661066674370 2023-10-05T19:41:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:3C2970 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.171+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=1B3918 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:425BED state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 10 manifest: total_size = 399144 2023-10-05T19:41:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.14638418041699668606 2023-10-05T19:41:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:425BED state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.179+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=3C2970 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:58ED7D state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 10 manifest: total_size = 49265 2023-10-05T19:41:16.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.10429446800311712628 2023-10-05T19:41:16.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:58ED7D state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.187+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=425BED 2023-10-05T19:41:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:59B6CE state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 10 manifest: total_size = 399212 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.11764093146047644228 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:59B6CE state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.195+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.205 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.203+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=58ED7D 2023-10-05T19:41:16.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:EBE004 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 10 manifest: total_size = 399250 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.3566384387499453885 2023-10-05T19:41:16.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:EBE004 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.207+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.211+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=59B6CE 2023-10-05T19:41:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:F39F63 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 10 manifest: total_size = 642824 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.9374760021168409484 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=test:F39F63 state=0x55fd02487218 s->prefetch_data=0 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd02487218 2023-10-05T19:41:16.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.215+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518.10 2023-10-05T19:41:16.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.219+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=EBE004 2023-10-05T19:41:16.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:41:16.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:41:16.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.227+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:41:16.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb560f5a700 20 handle_completion(): completion ok for obj=F39F63 2023-10-05T19:41:16.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:41:16.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:41:16.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:41:16.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:41:16.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518])"; uploads.size()=0; is_truncated=0 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]) 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 lifecycle: get_system_obj_state: rctx=0x7ffd9300af68 obj=default.rgw.meta:root:test state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-10-05T19:41:16.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-10-05T19:41:16.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.231+0000 7fb565429e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 info.flags=0x17 2023-10-05T19:41:16.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 to cache LRU end 2023-10-05T19:41:16.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518, ofs0, ns] 2023-10-05T19:41:16.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758808 cookie 94545152266944 notifier 4247 bl.length()=800 2023-10-05T19:41:16.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb560f5a700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 info.flags=0x17 2023-10-05T19:41:16.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb560f5a700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 to cache LRU end 2023-10-05T19:41:16.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb560f5a700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]): orig_sources= new_sources= 2023-10-05T19:41:16.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]): potential sources added=[] removed=[] 2023-10-05T19:41:16.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]): orig_dests= new_dests= 2023-10-05T19:41:16.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.235+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:test[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518]): potential dests added=[] removed=[] 2023-10-05T19:41:16.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 20 get_system_obj_state: rctx=0x7ffd9300a9e8 obj=default.rgw.meta:root:test state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:16.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-10-05T19:41:16.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 moving default.rgw.meta+root+test to cache LRU end 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 removing default.rgw.meta+root+test from cache 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-10-05T19:41:16.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.247+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758692 cookie 94545152256096 notifier 4247 bl.length()=148 2023-10-05T19:41:16.258 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.255+0000 7fb565429e40 10 removing default.rgw.meta+root+.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 from cache 2023-10-05T19:41:16.258 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.255+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518, ofs0, ns] 2023-10-05T19:41:16.259 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.255+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758809 cookie 94545152266944 notifier 4247 bl.length()=207 2023-10-05T19:41:16.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.275+0000 7fb565429e40 10 removing default.rgw.log++pubsub..bucket.test/caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518 from cache 2023-10-05T19:41:16.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.275+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/caa95e63-d85f-4735-a9cf-10d220ce9576.4207.518, ofs0, ns] 2023-10-05T19:41:16.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.275+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758693 cookie 94545152256096 notifier 4247 bl.length()=204 2023-10-05T19:41:16.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 20 get_system_obj_state: rctx=0x7ffd9300b4b0 obj=default.rgw.meta:root:.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:16.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 : miss 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 info.flags=0x16 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 to cache LRU end 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 20 Read xattr: user.rgw.acl 2023-10-05T19:41:16.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:41:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.279+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T19:41:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 info.flags=0x13 2023-10-05T19:41:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 to cache LRU end 2023-10-05T19:41:16.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 2023-10-05T19:41:16.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.283+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:41:16.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.287+0000 7fb565429e40 10 cls_bucket_list_unordered: got B02C68 2023-10-05T19:41:16.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.287+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=testbuck2:B02C68 state=0x55fd024846b8 s->prefetch_data=0 2023-10-05T19:41:16.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.291+0000 7fb565429e40 10 manifest: total_size = 399052 2023-10-05T19:41:16.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.291+0000 7fb565429e40 20 get_obj_state: setting s->obj_tag to caa95e63-d85f-4735-a9cf-10d220ce9576.4205.3146085030634536669 2023-10-05T19:41:16.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.291+0000 7fb565429e40 20 get_obj_state: rctx=0x7ffd9300b4d0 obj=testbuck2:B02C68 state=0x55fd024846b8 s->prefetch_data=0 2023-10-05T19:41:16.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.291+0000 7fb565429e40 20 prepare_atomic_modification: state is not atomic. state=0x55fd024846b8 2023-10-05T19:41:16.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.291+0000 7fb565429e40 20 bucket index object: default.rgw.buckets.index:.dir.caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519.7 2023-10-05T19:41:16.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-10-05T19:41:16.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-10-05T19:41:16.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-10-05T19:41:16.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:41:16.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:41:16.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:41:16.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:41:16.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:41:16.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb565429e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:41:16.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.295+0000 7fb55bfff700 20 handle_completion(): completion ok for obj=B02C68 2023-10-05T19:41:16.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:41:16.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:41:16.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:41:16.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:41:16.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519])"; uploads.size()=0; is_truncated=0 2023-10-05T19:41:16.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]) 2023-10-05T19:41:16.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 lifecycle: get_system_obj_state: rctx=0x7ffd9300af68 obj=default.rgw.meta:root:testbuck2 state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:16.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:16.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:41:16.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-10-05T19:41:16.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-10-05T19:41:16.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:41:16.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.299+0000 7fb565429e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 info.flags=0x17 2023-10-05T19:41:16.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 to cache LRU end 2023-10-05T19:41:16.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519, ofs0, ns] 2023-10-05T19:41:16.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb55bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758810 cookie 94545152266944 notifier 4247 bl.length()=810 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb55bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 info.flags=0x17 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb55bfff700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 to cache LRU end 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb55bfff700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]): orig_sources= new_sources= 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]): potential sources added=[] removed=[] 2023-10-05T19:41:16.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]): orig_dests= new_dests= 2023-10-05T19:41:16.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.303+0000 7fb565429e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519]): potential dests added=[] removed=[] 2023-10-05T19:41:16.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 20 get_system_obj_state: rctx=0x7ffd9300a9e8 obj=default.rgw.meta:root:testbuck2 state=0x55fd0220f0b0 s->prefetch_data=0 2023-10-05T19:41:16.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 removing default.rgw.meta+root+testbuck2 from cache 2023-10-05T19:41:16.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-10-05T19:41:16.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.315+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758694 cookie 94545152254464 notifier 4247 bl.length()=153 2023-10-05T19:41:16.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.319+0000 7fb565429e40 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 from cache 2023-10-05T19:41:16.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.319+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519, ofs0, ns] 2023-10-05T19:41:16.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.319+0000 7fb55bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758811 cookie 94545152266944 notifier 4247 bl.length()=212 2023-10-05T19:41:16.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.343+0000 7fb565429e40 10 removing default.rgw.log++pubsub..bucket.testbuck2/caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519 from cache 2023-10-05T19:41:16.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.343+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/caa95e63-d85f-4735-a9cf-10d220ce9576.4207.519, ofs0, ns] 2023-10-05T19:41:16.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.347+0000 7fb55bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758695 cookie 94545152256096 notifier 4247 bl.length()=209 2023-10-05T19:41:16.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.351+0000 7fb565429e40 10 removing key index: BHALCIPGHFAQINSURGKR 2023-10-05T19:41:16.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.351+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:BHALCIPGHFAQINSURGKR, ofs0, ns] 2023-10-05T19:41:16.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.351+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758696 cookie 94545152247696 notifier 4247 bl.length()=170 2023-10-05T19:41:16.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb565429e40 10 removing email index: foo.client.0+test@test.test 2023-10-05T19:41:16.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T19:41:16.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb55bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758697 cookie 94545152256096 notifier 4247 bl.length()=178 2023-10-05T19:41:16.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb565429e40 10 removing user buckets index 2023-10-05T19:41:16.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-10-05T19:41:16.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.355+0000 7fb560f5a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758698 cookie 94545152247696 notifier 4247 bl.length()=169 2023-10-05T19:41:16.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.359+0000 7fb565429e40 10 removing user index: foo.client.0 2023-10-05T19:41:16.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.359+0000 7fb565429e40 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-10-05T19:41:16.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.359+0000 7fb565429e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:41:16.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.363+0000 7fb55bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758812 cookie 94545152245280 notifier 4247 bl.length()=161 2023-10-05T19:41:16.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.367+0000 7fb565429e40 20 remove_watcher() i=0 2023-10-05T19:41:16.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.367+0000 7fb565429e40 2 removed watcher, disabling cache 2023-10-05T19:41:16.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.371+0000 7fb565429e40 20 remove_watcher() i=1 2023-10-05T19:41:16.376 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.371+0000 7fb565429e40 20 remove_watcher() i=2 2023-10-05T19:41:16.378 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.375+0000 7fb565429e40 20 remove_watcher() i=3 2023-10-05T19:41:16.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.375+0000 7fb565429e40 20 remove_watcher() i=4 2023-10-05T19:41:16.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.379+0000 7fb565429e40 20 remove_watcher() i=5 2023-10-05T19:41:16.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.383+0000 7fb565429e40 20 remove_watcher() i=6 2023-10-05T19:41:16.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.383+0000 7fb565429e40 20 remove_watcher() i=7 2023-10-05T19:41:16.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.387+0000 7fb4affff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:16.400 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-10-05T19:41:16.442 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:41:16.442 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:41:16.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.451+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 realm 2023-10-05T19:41:16.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:41:16.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:16.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.455+0000 7f4189309e40 20 searching for the correct realm 2023-10-05T19:41:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.479+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:16.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.483+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:16.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:16.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:16.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:16.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:16.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.507+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.514 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:16.514 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 zone default found 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 4 Realm: () 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:41:16.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:41:16.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 zonegroup default 2023-10-05T19:41:16.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.511+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.515+0000 7f4189309e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.515+0000 7f4189309e40 20 started sync module instance, tier type = 2023-10-05T19:41:16.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.515+0000 7f4189309e40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:41:16.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.531+0000 7f4189309e40 20 add_watcher() i=0 2023-10-05T19:41:16.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.531+0000 7f4189309e40 20 add_watcher() i=1 2023-10-05T19:41:16.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.531+0000 7f4189309e40 20 add_watcher() i=2 2023-10-05T19:41:16.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.531+0000 7f4189309e40 20 add_watcher() i=3 2023-10-05T19:41:16.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.531+0000 7f4189309e40 20 add_watcher() i=4 2023-10-05T19:41:16.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.535+0000 7f4189309e40 20 add_watcher() i=5 2023-10-05T19:41:16.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.535+0000 7f4189309e40 20 add_watcher() i=6 2023-10-05T19:41:16.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.535+0000 7f4189309e40 20 add_watcher() i=7 2023-10-05T19:41:16.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.535+0000 7f4189309e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:41:16.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.539+0000 7f40d3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:16.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.539+0000 7f4189309e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:41:16.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.539+0000 7f4189309e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:41:16.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.643+0000 7f40c2ffd700 20 reqs_thread_entry: start 2023-10-05T19:41:16.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.643+0000 7f40c0ff9700 20 reqs_thread_entry: start 2023-10-05T19:41:16.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.643+0000 7f4189309e40 20 init_complete bucket index max shards: 11 2023-10-05T19:41:16.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.643+0000 7f4189309e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:41:16.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f40ba7fc700 20 reqs_thread_entry: start 2023-10-05T19:41:16.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 20 get_system_obj_state: rctx=0x7fff33ed3038 obj=default.rgw.meta:users.uid:bar.client.0 state=0x56144ea970b0 s->prefetch_data=0 2023-10-05T19:41:16.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f40baffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:41:47 2023 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:41:16.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 removing key index: AGVUHNNTZZVPWUCVHGBJ 2023-10-05T19:41:16.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.647+0000 7f4189309e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:AGVUHNNTZZVPWUCVHGBJ, ofs0, ns] 2023-10-05T19:41:16.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.651+0000 7f4184e3a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757532 cookie 94645220530416 notifier 4253 bl.length()=170 2023-10-05T19:41:16.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.651+0000 7f4189309e40 10 removing email index: bar.client.0+test@test.test 2023-10-05T19:41:16.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.651+0000 7f4189309e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T19:41:16.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.655+0000 7f417ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758699 cookie 94645220520224 notifier 4253 bl.length()=178 2023-10-05T19:41:16.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.659+0000 7f4189309e40 10 removing user buckets index 2023-10-05T19:41:16.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.659+0000 7f4189309e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-10-05T19:41:16.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.659+0000 7f417ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208757533 cookie 94645220522432 notifier 4253 bl.length()=169 2023-10-05T19:41:16.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.663+0000 7f4189309e40 10 removing user index: bar.client.0 2023-10-05T19:41:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.663+0000 7f4189309e40 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-10-05T19:41:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.663+0000 7f4189309e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:41:16.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.663+0000 7f4184e3a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758700 cookie 94645220520224 notifier 4253 bl.length()=161 2023-10-05T19:41:16.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.667+0000 7f4189309e40 20 remove_watcher() i=0 2023-10-05T19:41:16.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.667+0000 7f4189309e40 2 removed watcher, disabling cache 2023-10-05T19:41:16.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.671+0000 7f4189309e40 20 remove_watcher() i=1 2023-10-05T19:41:16.676 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.671+0000 7f4189309e40 20 remove_watcher() i=2 2023-10-05T19:41:16.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.679+0000 7f4189309e40 20 remove_watcher() i=3 2023-10-05T19:41:16.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.679+0000 7f4189309e40 20 remove_watcher() i=4 2023-10-05T19:41:16.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.683+0000 7f4189309e40 20 remove_watcher() i=5 2023-10-05T19:41:16.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.683+0000 7f4189309e40 20 remove_watcher() i=6 2023-10-05T19:41:16.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.687+0000 7f4189309e40 20 remove_watcher() i=7 2023-10-05T19:41:16.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.691+0000 7f40d3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:16.706 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-10-05T19:41:16.747 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:41:16.747 INFO:teuthology.orchestra.run.smithi064.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:41:16.758 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 realm 2023-10-05T19:41:16.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.761 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:41:16.761 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.761 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.761 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.755+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.759+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.759+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.759+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:16.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.759+0000 7f8b562ede40 20 searching for the correct realm 2023-10-05T19:41:16.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:16.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:16.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:16.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:16.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:16.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.783+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.787+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zonegroup_info.e8f4181e-bff1-44b7-bdfa-eded3e62e879 2023-10-05T19:41:16.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zone_info.caa95e63-d85f-4735-a9cf-10d220ce9576 2023-10-05T19:41:16.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:41:16.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:41:16.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:41:16.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:41:16.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.811+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:41:16.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 zone default found 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 4 Realm: () 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 4 ZoneGroup: default (e8f4181e-bff1-44b7-bdfa-eded3e62e879) 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 4 Zone: default (caa95e63-d85f-4735-a9cf-10d220ce9576) 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:41:16.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 zonegroup default 2023-10-05T19:41:16.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:41:16.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 started sync module instance, tier type = 2023-10-05T19:41:16.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.815+0000 7f8b562ede40 20 started zone id=caa95e63-d85f-4735-a9cf-10d220ce9576 (name=default) with tier type = 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=0 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=1 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=2 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=3 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=4 2023-10-05T19:41:16.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=5 2023-10-05T19:41:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.839+0000 7f8b562ede40 20 add_watcher() i=6 2023-10-05T19:41:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.843+0000 7f8b562ede40 20 add_watcher() i=7 2023-10-05T19:41:16.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.843+0000 7f8b562ede40 2 all 8 watchers are set, enabling cache 2023-10-05T19:41:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.843+0000 7f8b257e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.843+0000 7f8b562ede40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:41:16.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.843+0000 7f8b562ede40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:41:16.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.943+0000 7f8a93fff700 20 reqs_thread_entry: start 2023-10-05T19:41:16.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8a91ffb700 20 reqs_thread_entry: start 2023-10-05T19:41:16.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 init_complete bucket index max shards: 11 2023-10-05T19:41:16.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:41:16.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8a7ffff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:41:47 2023 2023-10-05T19:41:16.950 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-05T19:41:16.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8a7f7fe700 20 reqs_thread_entry: start 2023-10-05T19:41:16.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 get_system_obj_state: rctx=0x7fff74f9dfa8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x556758eea0b0 s->prefetch_data=0 2023-10-05T19:41:16.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:41:16.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T19:41:16.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:41:16.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:41:16.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 Read xattr: user.rgw.idtag 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 rados->read ofs=0 len=0 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 20 rados_obj.operate() r=0 bl.length=528 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:41:16.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:41:16.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 removing key index: WXHJGDAHUYMPQZHZNRMV 2023-10-05T19:41:16.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.947+0000 7f8b562ede40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:WXHJGDAHUYMPQZHZNRMV, ofs0, ns] 2023-10-05T19:41:16.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.951+0000 7f8b5161d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758813 cookie 93902363488016 notifier 4259 bl.length()=170 2023-10-05T19:41:16.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.955+0000 7f8b562ede40 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-10-05T19:41:16.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.955+0000 7f8b562ede40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T19:41:16.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.955+0000 7f8b51e1e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758814 cookie 93902363490304 notifier 4259 bl.length()=193 2023-10-05T19:41:16.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.959+0000 7f8b562ede40 10 removing user buckets index 2023-10-05T19:41:16.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.959+0000 7f8b562ede40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-10-05T19:41:16.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.959+0000 7f8b5161d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758701 cookie 93902363485296 notifier 4259 bl.length()=184 2023-10-05T19:41:16.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.963+0000 7f8b562ede40 10 removing user index: testx$tenanteduser.client.0 2023-10-05T19:41:16.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.963+0000 7f8b562ede40 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-10-05T19:41:16.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.963+0000 7f8b562ede40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:41:16.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.963+0000 7f8b51e1e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 163208758815 cookie 93902363488016 notifier 4259 bl.length()=176 2023-10-05T19:41:16.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.967+0000 7f8b562ede40 20 remove_watcher() i=0 2023-10-05T19:41:16.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.971+0000 7f8b562ede40 2 removed watcher, disabling cache 2023-10-05T19:41:16.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.971+0000 7f8b562ede40 20 remove_watcher() i=1 2023-10-05T19:41:16.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.975+0000 7f8b562ede40 20 remove_watcher() i=2 2023-10-05T19:41:16.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.979+0000 7f8b562ede40 20 remove_watcher() i=3 2023-10-05T19:41:16.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.987+0000 7f8b562ede40 20 remove_watcher() i=4 2023-10-05T19:41:16.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.987+0000 7f8b562ede40 20 remove_watcher() i=5 2023-10-05T19:41:16.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.991+0000 7f8b562ede40 20 remove_watcher() i=6 2023-10-05T19:41:16.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.995+0000 7f8b562ede40 20 remove_watcher() i=7 2023-10-05T19:41:17.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-05T19:41:16.995+0000 7f8b257e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:41:17.012 INFO:tasks.s3tests:Removing s3-tests... 2023-10-05T19:41:17.012 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:41:17.170 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-10-05T19:41:17.179 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T19:41:17.198 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T19:41:17.198 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:17.199 INFO:tasks.rgw.client.0.smithi064.stdout:2023-10-05T19:41:17.195+0000 7f3f3dc14700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.64:5696 --foreground (PID: 33857) UID: 0 2023-10-05T19:41:17.199 INFO:tasks.rgw.client.0.smithi064.stdout:2023-10-05T19:41:17.195+0000 7f3f420f4e40 -1 shutting down 2023-10-05T19:41:22.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T19:41:23.299 INFO:tasks.rgw.client.0:Stopped 2023-10-05T19:41:23.300 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T19:41:23.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T19:41:23.363 DEBUG:teuthology.run_tasks:Unwinding manager pykmip 2023-10-05T19:41:23.373 INFO:tasks.pykmip:Stopping PyKMIP instance 2023-10-05T19:41:23.374 DEBUG:tasks.pykmip.client.0:waiting for process to exit 2023-10-05T19:41:23.374 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:28.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:54: CryptographyDeprecationWarning: Blowfish has been deprecated 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr: enums.CryptographicAlgorithm.BLOWFISH: algorithms.Blowfish, 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:56: CryptographyDeprecationWarning: CAST5 has been deprecated 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr: enums.CryptographicAlgorithm.CAST5: algorithms.CAST5, 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:57: CryptographyDeprecationWarning: IDEA has been deprecated 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr: enums.CryptographicAlgorithm.IDEA: algorithms.IDEA, 2023-10-05T19:41:28.664 INFO:tasks.pykmip.client.0.smithi064.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/engine.py:438: SAWarning: TypeDecorator EnumType() will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this warning at: https://sqlalche.me/e/20/cprf) 2023-10-05T19:41:28.665 INFO:tasks.pykmip.client.0.smithi064.stderr: object_type = self._data_session.query( 2023-10-05T19:41:28.784 INFO:tasks.pykmip.client.0:Stopped 2023-10-05T19:41:28.784 INFO:tasks.pykmip:Removing system dependencies of PyKMIP... 2023-10-05T19:41:28.784 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libffi-dev 2023-10-05T19:41:28.859 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:41:29.068 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:41:29.069 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:41:29.254 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:41:29.255 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:41:29.255 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:41:29.281 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be REMOVED: 2023-10-05T19:41:29.282 INFO:teuthology.orchestra.run.smithi064.stdout: libffi-dev* 2023-10-05T19:41:29.653 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 1 to remove and 249 not upgraded. 2023-10-05T19:41:29.654 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 312 kB disk space will be freed. 2023-10-05T19:41:29.773 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153030 files and directories currently installed.) 2023-10-05T19:41:29.777 INFO:teuthology.orchestra.run.smithi064.stdout:Removing libffi-dev:amd64 (3.3-4) ... 2023-10-05T19:41:30.361 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-10-05T19:41:30.686 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:41:32.363 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libsqlite3-dev 2023-10-05T19:41:32.501 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:41:32.712 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:41:32.743 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:41:32.922 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:41:32.923 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:41:32.923 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:41:32.954 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be REMOVED: 2023-10-05T19:41:32.954 INFO:teuthology.orchestra.run.smithi064.stdout: libsqlite3-dev* 2023-10-05T19:41:33.335 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 1 to remove and 249 not upgraded. 2023-10-05T19:41:33.336 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 2,372 kB disk space will be freed. 2023-10-05T19:41:33.458 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152996 files and directories currently installed.) 2023-10-05T19:41:33.463 INFO:teuthology.orchestra.run.smithi064.stdout:Removing libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-10-05T19:41:34.352 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libssl-dev 2023-10-05T19:41:34.425 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:41:34.633 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:41:34.634 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:41:34.831 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:41:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:41:34.832 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:41:34.860 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be REMOVED: 2023-10-05T19:41:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: libssl-dev* 2023-10-05T19:41:35.223 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 1 to remove and 249 not upgraded. 2023-10-05T19:41:35.223 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 8,010 kB disk space will be freed. 2023-10-05T19:41:35.388 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152987 files and directories currently installed.) 2023-10-05T19:41:35.393 INFO:teuthology.orchestra.run.smithi064.stdout:Removing libssl-dev:amd64 (1.1.1f-1ubuntu2.19) ... 2023-10-05T19:41:36.373 DEBUG:teuthology.orchestra.run.smithi064:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge python-dev 2023-10-05T19:41:37.753 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-05T19:41:37.973 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-05T19:41:37.974 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-05T19:41:38.159 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:41:38.160 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:41:38.160 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:41:38.212 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 249 not upgraded. 2023-10-05T19:41:38.214 INFO:tasks.pykmip:Removing pykmip... 2023-10-05T19:41:38.215 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/pykmip 2023-10-05T19:41:38.281 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2023-10-05T19:41:38.291 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmiproot.key 2023-10-05T19:41:38.295 DEBUG:teuthology.orchestra.run.smithi064:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-10-05T19:41:38.312 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/ca/kmip-client.crt /home/ubuntu/cephtest/ca/kmip-client.key 2023-10-05T19:41:38.318 DEBUG:teuthology.orchestra.run.smithi064:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-10-05T19:41:38.324 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/ca/kmip-server.crt /home/ubuntu/cephtest/ca/kmip-server.key 2023-10-05T19:41:38.330 DEBUG:teuthology.orchestra.run.smithi064:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-10-05T19:41:38.336 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T19:41:38.345 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T19:41:38.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:41:38.669 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-05T19:41:38.669 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":38,"fsid":"c440a5a2-ff86-49f1-ad52-251a03eaf475","created":"2023-10-05T19:34:22.746659+0000","modified":"2023-10-05T19:37:29.765753+0000","last_up_change":"2023-10-05T19:34:31.007994+0000","last_in_change":"2023-10-05T19:34:23.569516+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:34:31.195305+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:34:37.822834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T19:35:27.862483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T19:35:30.131224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T19:35:32.149950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T19:35:33.929529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T19:35:35.877318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T19:35:38.022001+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-10-05T19:35:48.338455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-10-05T19:37:10.186208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"b96e8ea8-4da5-46ed-b092-decf44b64b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":29056},{"type":"v1","addr":"172.21.15.64:6817","nonce":29056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":29056},{"type":"v1","addr":"172.21.15.64:6819","nonce":29056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":29056},{"type":"v1","addr":"172.21.15.64:6823","nonce":29056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":29056},{"type":"v1","addr":"172.21.15.64:6821","nonce":29056}]},"public_addr":"172.21.15.64:6817/29056","cluster_addr":"172.21.15.64:6819/29056","heartbeat_back_addr":"172.21.15.64:6823/29056","heartbeat_front_addr":"172.21.15.64:6821/29056","state":["exists","up"]},{"osd":1,"uuid":"4bfb468e-1104-450b-8b9c-be1b059ffc68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":29055},{"type":"v1","addr":"172.21.15.64:6809","nonce":29055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":29055},{"type":"v1","addr":"172.21.15.64:6811","nonce":29055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":29055},{"type":"v1","addr":"172.21.15.64:6815","nonce":29055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":29055},{"type":"v1","addr":"172.21.15.64:6813","nonce":29055}]},"public_addr":"172.21.15.64:6809/29055","cluster_addr":"172.21.15.64:6811/29055","heartbeat_back_addr":"172.21.15.64:6815/29055","heartbeat_front_addr":"172.21.15.64:6813/29055","state":["exists","up"]},{"osd":2,"uuid":"2128b6c4-0dba-4886-9d99-0513312ced44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":29054},{"type":"v1","addr":"172.21.15.64:6801","nonce":29054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":29054},{"type":"v1","addr":"172.21.15.64:6803","nonce":29054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":29054},{"type":"v1","addr":"172.21.15.64:6807","nonce":29054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":29054},{"type":"v1","addr":"172.21.15.64:6805","nonce":29054}]},"public_addr":"172.21.15.64:6801/29054","cluster_addr":"172.21.15.64:6803/29054","heartbeat_back_addr":"172.21.15.64:6807/29054","heartbeat_front_addr":"172.21.15.64:6805/29054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.314369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.274631+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:34:28.128680+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.20","mappings":[{"from":0,"to":1}]},{"pgid":"3.2e","mappings":[{"from":0,"to":1}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"4.34","mappings":[{"from":1,"to":0}]},{"pgid":"4.35","mappings":[{"from":1,"to":2}]},{"pgid":"10.0","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:41:38.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T19:41:39.002 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T19:41:39.003 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T19:41:39.003 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T19:41:39.003 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:39.100 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T19:41:39.101 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T19:41:39.101 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:39.174 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T19:41:39.174 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T19:41:39.174 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:39.246 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T19:41:39.246 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T19:41:39.246 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T19:41:39.246 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:39.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-05T19:41:39.277 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T19:41:39.277 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T19:41:39.277 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T19:41:39.277 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:41:39.291 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T19:41:39.291 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T19:41:39.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:41:39.351 INFO:teuthology.orchestra.run.smithi064.stdout:1696534479.7151935 mon.a (mon.0) 95 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:41:39.352 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T19:41:39.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:41:39.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:41:39.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:41:39.442 INFO:teuthology.orchestra.run.smithi064.stdout:1696534479.7151935 mon.a (mon.0) 95 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:41:39.443 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:41:39.443 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T19:41:40.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:41:40.318 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T19:41:40.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:41:40.846 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T19:41:41.071 INFO:tasks.ceph:Archiving mon data... 2023-10-05T19:41:41.073 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752/data/mon.a.tgz 2023-10-05T19:41:41.073 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-10-05T19:41:41.079 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.G5wU8CR2nN 2023-10-05T19:41:41.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.G5wU8CR2nN 2023-10-05T19:41:41.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.G5wU8CR2nN 2023-10-05T19:41:41.239 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.G5wU8CR2nN is 325KB 2023-10-05T19:41:41.299 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.G5wU8CR2nN 2023-10-05T19:41:41.306 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T19:41:41.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T19:41:41.320 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T19:41:41.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T19:41:41.461 INFO:teuthology.orchestra.run.smithi064.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T19:41:41.469 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T19:41:41.470 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752/remote/smithi064/crash 2023-10-05T19:41:41.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T19:41:41.486 INFO:tasks.ceph:Compressing logs... 2023-10-05T19:41:41.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:41:44.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-05T19:42:10.691 INFO:tasks.ceph:Archiving logs... 2023-10-05T19:42:10.691 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752/remote/smithi064/log 2023-10-05T19:42:10.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T19:42:14.176 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T19:42:14.186 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T19:42:14.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T19:42:14.246 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T19:42:14.256 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T19:42:14.256 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout:*hv01.front.sepi 44.4.53.2 2 u 18 64 377 0.073 1.365 0.777 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 64 377 0.072 0.075 1.092 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout:+hv03.front.sepi 74.6.168.72 3 u 19 64 377 0.093 -0.790 0.779 2023-10-05T19:42:14.280 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 0.000 0.000 2023-10-05T19:42:14.281 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T19:42:14.290 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T19:42:14.291 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T19:42:14.300 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T19:42:14.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T19:42:14.318 INFO:teuthology.task.internal:Duration was 964.285628 seconds 2023-10-05T19:42:14.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T19:42:14.327 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T19:42:14.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T19:42:14.380 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T19:42:14.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2023-10-05T19:42:14.380 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T19:42:14.393 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T19:42:14.393 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:42:14.419 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T19:42:14.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T19:42:14.480 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T19:42:14.491 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T19:42:14.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T19:42:14.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T19:42:14.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:42:14.528 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2023-10-05T19:42:14.543 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:42:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:42:14.590 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T19:42:14.601 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T19:42:14.602 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752/remote/smithi064 2023-10-05T19:42:14.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T19:42:14.624 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T19:42:14.624 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T19:42:14.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T19:42:14.677 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T19:42:14.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T19:42:14.687 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T19:42:14.688 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T19:42:14.695 INFO:teuthology.orchestra.run.smithi064.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 19:42 /home/ubuntu/cephtest 2023-10-05T19:42:14.698 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T19:42:14.709 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T19:42:14.731 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T19:42:14.756 DEBUG:teuthology.nuke:shortname: smithi064 2023-10-05T19:42:14.757 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:42:14.779 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:17:00.414337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-10-05T19:42:14.828 INFO:teuthology.orchestra.console.smithi064:Power off 2023-10-05T19:42:14.828 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T19:42:14.854 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2023-10-05T19:42:14.854 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:42:14.875 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-10-05T19:42:18.877 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:42:18.989 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-10-05T19:42:22.990 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:42:23.099 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2023-10-05T19:42:23.200 INFO:teuthology.orchestra.console.smithi064:Power off completed 2023-10-05T19:42:23.304 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} duration: 964.2856276035309 failure_reason: '"1696534479.7151935 mon.a (mon.0) 95 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T19:42:23.305 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:42:23.388 INFO:teuthology.run:FAIL