2023-10-05T19:26:02.694 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:26:02.695 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757 --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757/orig.config.yaml 2023-10-05T19:26:02.723 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:26:02.791 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7411757' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption_keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqsJPXSyQ8/UoKTpMll+bKPWbkWz3arfKZaC6WX3RqXyZCaa/eZYDhqtXH65cBInWzNcuT84sUvnATNTf/gbwA= tasks: - install: null - ceph: null - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:26:02.791 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:26:02.792 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:26:02.792 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:26:02.794 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:26:02.815 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:26:02.815 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:26:02.815 INFO:teuthology.packaging:ref: None 2023-10-05T19:26:02.815 INFO:teuthology.packaging:tag: None 2023-10-05T19:26:02.815 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:26:02.816 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:26:02.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:26:02.962 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278.g487ffcb5 2023-10-05T19:26:02.962 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:26:02.969 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:26:02.969 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:26:02.981 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:26:02.992 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:26:03.000 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:26:03.023 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:20:58.448078', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqsJPXSyQ8/UoKTpMll+bKPWbkWz3arfKZaC6WX3RqXyZCaa/eZYDhqtXH65cBInWzNcuT84sUvnATNTf/gbwA='} 2023-10-05T19:26:03.023 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:26:03.029 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-05T19:26:03.030 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:26:03.070 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5e6ce90ee0>, signals=[15]) 2023-10-05T19:26:03.070 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:26:03.077 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:26:03.077 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T19:26:03.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:26:03.144 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:26:03.151 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-10-05T19:26:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-10-05T19:26:03.216 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-10-05T19:26:03.276 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2023-10-05T19:26:03.277 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-05T19:26:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-05T19:26:03.278 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-05T19:26:03.278 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-10-05T19:26:03.305 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:26:03.315 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:26:03.321 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:26:03.321 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:26:03.341 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:26:03.348 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:26:03.348 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:26:03.406 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:26:03.440 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:26:03.454 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:26:03.454 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:26:03.455 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:26:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:26:03.475 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-10-05T19:26:03.535 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-489.el8.x86_64 2023-10-05T19:26:03.535 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-489.el8.x86_64 2023-10-05T19:26:03.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2023-10-05T19:36:10.263 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 57 kB/s | 33 MB 10:04 2023-10-05T19:36:17.146 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 68 MB/s | 48 MB 00:00 2023-10-05T19:36:24.271 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 56 kB/s | 18 kB 00:00 2023-10-05T19:36:24.630 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 38 kB/s | 6.9 kB 00:00 2023-10-05T19:36:24.915 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-05T19:36:26.749 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2023-10-05T19:36:30.507 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 388 kB/s | 24 kB 00:00 2023-10-05T19:36:33.948 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-05T19:36:33.948 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-05T19:36:34.204 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:36:34.205 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T19:36:34.206 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T19:36:34.206 INFO:teuthology.orchestra.run.smithi086.stdout:Install 3 Packages 2023-10-05T19:36:34.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:36:34.206 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 88 M 2023-10-05T19:36:34.206 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2023-10-05T19:36:34.207 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T19:36:34.492 INFO:teuthology.orchestra.run.smithi086.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 41 MB/s | 10 MB 00:00 2023-10-05T19:36:34.893 INFO:teuthology.orchestra.run.smithi086.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 55 MB/s | 35 MB 00:00 2023-10-05T19:36:35.284 INFO:teuthology.orchestra.run.smithi086.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 41 MB/s | 43 MB 00:01 2023-10-05T19:36:35.284 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:36:35.284 INFO:teuthology.orchestra.run.smithi086.stdout:Total 82 MB/s | 88 MB 00:01 2023-10-05T19:36:35.948 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T19:36:36.370 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T19:36:36.370 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T19:36:37.584 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T19:36:37.603 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T19:36:40.365 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T19:36:40.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:36:44.496 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:36:44.904 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:36:48.284 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:36:48.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:37:10.596 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:37:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:37:11.383 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:37:11.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-05T19:37:11.383 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-05T19:37:12.126 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:37:12.127 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T19:37:12.279 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:37:12.497 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-05T19:37:12.497 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-05T19:37:12.497 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-05T19:37:12.497 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:37:12.497 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:37:12.497 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:37:12.497 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:37:12.498 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:37:12.704 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-05T19:37:12.705 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-05T19:37:12.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2023-10-05T19:37:12.738 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2023-10-05T19:37:12.738 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2023-10-05T19:37:12.738 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 129.el8 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9265586 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-05T19:37:12.739 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : CentOS BuildSystem 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2023-10-05T19:37:12.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T19:37:12.741 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2023-10-05T19:37:12.741 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-05T19:37:12.741 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-514.el8.x86_64 2023-10-05T19:37:12.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-05T19:37:14.161 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2023-10-05T19:37:16.340 INFO:teuthology.orchestra.run.smithi086.stderr:done 2023-10-05T19:37:16.349 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-10-05T19:37:16.383 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.eam0K30NPG 2023-10-05T19:37:16.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eam0K30NPG 2023-10-05T19:37:16.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.eam0K30NPG 2023-10-05T19:37:16.610 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.eam0K30NPG is 6KB 2023-10-05T19:37:16.659 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.eam0K30NPG 2023-10-05T19:37:16.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2023-10-05T19:37:16.750 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-05T19:37:16.750 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-05T19:37:16.750 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-05T19:37:16.750 INFO:teuthology.orchestra.run.smithi086.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-05T19:37:16.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-05T19:37:16.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2023-10-05T19:37:46.986 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:37:46.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-10-05T19:37:46.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:38:47.046 DEBUG:teuthology.orchestra.remote:timed out 2023-10-05T19:38:56.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-10-05T19:38:56.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:38:56.583 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-10-05T19:38:56.874 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-10-05T19:38:56.874 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-10-05T19:38:56.875 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-10-05T19:38:56.932 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-514.el8.x86_64 2023-10-05T19:38:56.940 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-05T19:38:56.940 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:38:56.940 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-05T19:38:57.942 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T19:38:58.314 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:38:58.388 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2023-10-05T19:38:58.496 DEBUG:teuthology.parallel:result is None 2023-10-05T19:38:58.497 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T19:38:58.508 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T19:38:58.509 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:38:58.528 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T19:38:58.705 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T19:38:59.649 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T19:38:59.765 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:38:59.795 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T19:38:59.802 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T19:38:59.802 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:38:59.885 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:38:59.947 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:38:59.948 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T19:38:59.955 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T19:38:59.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:38:59.985 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T19:39:00.095 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T19:39:00.095 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:39:00.113 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:39:00.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:39:00.277 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:39:00.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:39:00.430 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T19:39:01.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:39:01.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-10-05T19:39:01.456 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T19:39:01.618 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T19:39:01.648 INFO:teuthology.task.internal:Starting timer... 2023-10-05T19:39:01.648 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T19:39:01.675 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T19:39:01.740 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-05T19:39:01.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-10-05T19:39:01.824 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2023-10-05T19:39:01.826 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T19:39:01.826 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-10-05T19:39:01.890 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-10-05T19:39:01.890 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2023-10-05T19:39:01.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T19:39:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:39:01.952 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T19:39:01.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-10-05T19:39:02.021 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T19:39:02.031 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-05T19:39:02.036 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-05T19:39:02.149 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T19:39:02.165 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T19:39:02.167 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T19:55:35.966 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2023-10-05T19:55:35.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-10-05T19:55:35.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:55:36.035 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-10-05T19:55:36.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-10-05T19:55:36.105 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T19:55:36.118 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T19:55:36.118 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:55:36.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:55:36.177 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-05T19:55:36.190 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-05T19:55:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-10-05T19:55:36.271 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-10-05T19:55:36.285 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-05T19:55:36.299 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-05T19:55:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:55:36.379 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:55:36.380 INFO:teuthology.run_tasks:Running task install... 2023-10-05T19:55:36.388 DEBUG:teuthology.task.install:project ceph 2023-10-05T19:55:36.389 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T19:55:36.389 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T19:55:36.389 INFO:teuthology.task.install:Using flavor: default 2023-10-05T19:55:36.394 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T19:55:36.395 INFO:teuthology.task.install:extra packages: [] 2023-10-05T19:55:36.395 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'tag': None, 'wait_for_package': False} 2023-10-05T19:55:36.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:55:36.541 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/ 2023-10-05T19:55:36.542 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1278.g487ffcb5 2023-10-05T19:55:36.645 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-05T19:55:36.645 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T19:55:36.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-05T19:55:36.675 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-05T19:55:36.675 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/487ffcb5ddb60d29c5688d1a457e812b1b697b90/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-05T19:55:36.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-05T19:55:36.821 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-05T19:55:36.888 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-10-05T19:55:36.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-10-05T19:55:37.164 INFO:teuthology.orchestra.run.smithi086.stdout:76 files removed 2023-10-05T19:55:37.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2023-10-05T19:55:37.988 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 199 kB/s | 78 kB 00:00 2023-10-05T19:55:38.523 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 18 kB/s | 9.2 kB 00:00 2023-10-05T19:55:38.882 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 4.4 kB/s | 1.4 kB 00:00 2023-10-05T20:04:12.708 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - AppStream 66 kB/s | 33 MB 08:33 2023-10-05T20:04:19.633 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - BaseOS 54 MB/s | 48 MB 00:00 2023-10-05T20:04:26.866 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-10-05T20:04:27.250 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2023-10-05T20:04:29.126 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 8 - PowerTools 3.8 MB/s | 6.5 MB 00:01 2023-10-05T20:04:30.536 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-05T20:04:34.814 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 16 MB 00:04 2023-10-05T20:04:38.637 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 468 kB/s | 24 kB 00:00 2023-10-05T20:04:43.044 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:04:43.046 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:04:43.046 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.6 M 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.2 M 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 23 M 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 24 k 2023-10-05T20:04:43.047 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 797 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 506 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.6 M 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 45 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 115 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 191 k 2023-10-05T20:04:43.048 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 370 k 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 359 k 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 106 k 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout:Install 21 Packages 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2023-10-05T20:04:43.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:04:43.050 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 63 M 2023-10-05T20:04:43.050 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:04:44.371 INFO:teuthology.orchestra.run.smithi086.stdout:(1/23): ceph-base-17.2.6-1278.g487ffcb5.el8.x86 9.3 MB/s | 5.2 MB 00:00 2023-10-05T20:04:44.438 INFO:teuthology.orchestra.run.smithi086.stdout:(2/23): ceph-selinux-17.2.6-1278.g487ffcb5.el8. 366 kB/s | 24 kB 00:00 2023-10-05T20:04:44.538 INFO:teuthology.orchestra.run.smithi086.stdout:(3/23): libcephfs2-17.2.6-1278.g487ffcb5.el8.x8 7.8 MB/s | 797 kB 00:00 2023-10-05T20:04:44.622 INFO:teuthology.orchestra.run.smithi086.stdout:(4/23): libradosstriper1-17.2.6-1278.g487ffcb5. 5.9 MB/s | 506 kB 00:00 2023-10-05T20:04:45.114 INFO:teuthology.orchestra.run.smithi086.stdout:(5/23): librgw2-17.2.6-1278.g487ffcb5.el8.x86_6 11 MB/s | 5.6 MB 00:00 2023-10-05T20:04:45.272 INFO:teuthology.orchestra.run.smithi086.stdout:(6/23): ceph-radosgw-17.2.6-1278.g487ffcb5.el8. 12 MB/s | 17 MB 00:01 2023-10-05T20:04:45.289 INFO:teuthology.orchestra.run.smithi086.stdout:(7/23): python3-ceph-argparse-17.2.6-1278.g487f 255 kB/s | 45 kB 00:00 2023-10-05T20:04:45.314 INFO:teuthology.orchestra.run.smithi086.stdout:(8/23): python3-ceph-common-17.2.6-1278.g487ffc 2.7 MB/s | 115 kB 00:00 2023-10-05T20:04:45.373 INFO:teuthology.orchestra.run.smithi086.stdout:(9/23): python3-cephfs-17.2.6-1278.g487ffcb5.el 2.2 MB/s | 191 kB 00:00 2023-10-05T20:04:45.398 INFO:teuthology.orchestra.run.smithi086.stdout:(10/23): python3-rados-17.2.6-1278.g487ffcb5.el 4.3 MB/s | 370 kB 00:00 2023-10-05T20:04:45.424 INFO:teuthology.orchestra.run.smithi086.stdout:(11/23): python3-rbd-17.2.6-1278.g487ffcb5.el8. 7.0 MB/s | 359 kB 00:00 2023-10-05T20:04:45.449 INFO:teuthology.orchestra.run.smithi086.stdout:(12/23): python3-rgw-17.2.6-1278.g487ffcb5.el8. 2.1 MB/s | 106 kB 00:00 2023-10-05T20:04:48.133 INFO:teuthology.orchestra.run.smithi086.stdout:(13/23): ceph-common-17.2.6-1278.g487ffcb5.el8. 5.4 MB/s | 23 MB 00:04 2023-10-05T20:08:57.467 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-prettytable-0.7.2-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-prettytable-0.7.2-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:08:57.656 INFO:teuthology.orchestra.run.smithi086.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 180 B/s | 44 kB 04:09 2023-10-05T20:08:57.721 INFO:teuthology.orchestra.run.smithi086.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 726 kB/s | 47 kB 00:00 2023-10-05T20:08:57.963 INFO:teuthology.orchestra.run.smithi086.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-10-05T20:08:58.022 INFO:teuthology.orchestra.run.smithi086.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-10-05T20:08:58.080 INFO:teuthology.orchestra.run.smithi086.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-10-05T20:08:58.256 INFO:teuthology.orchestra.run.smithi086.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 9.9 MB/s | 1.7 MB 00:00 2023-10-05T20:08:59.381 INFO:teuthology.orchestra.run.smithi086.stdout:(20/23): librados2-17.2.6-1278.g487ffcb5.el8.x8 3.1 MB/s | 3.5 MB 00:01 2023-10-05T20:09:00.605 INFO:teuthology.orchestra.run.smithi086.stdout:(21/23): librbd1-17.2.6-1278.g487ffcb5.el8.x86_ 11 MB/s | 3.6 MB 00:00 2023-10-05T20:09:25.861 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] lttng-ust-2.8.1-11.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/lttng-ust-2.8.1-11.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:09:26.086 INFO:teuthology.orchestra.run.smithi086.stdout:(22/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 947 B/s | 259 kB 04:40 2023-10-05T20:10:17.453 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] librdkafka-0.11.4-3.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/librdkafka-0.11.4-3.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:10:18.073 INFO:teuthology.orchestra.run.smithi086.stdout:(23/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.1 kB/s | 354 kB 05:32 2023-10-05T20:10:18.073 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:10:18.073 INFO:teuthology.orchestra.run.smithi086.stdout:Total 194 kB/s | 63 MB 05:35 2023-10-05T20:10:18.113 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:10:18.158 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:10:18.159 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:10:18.533 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:10:18.536 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:10:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:10:19.923 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-10-05T20:10:20.115 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-05T20:10:20.485 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-05T20:10:20.641 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:10:20.768 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:10:21.023 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T20:10:21.190 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:10:21.630 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:10:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T20:10:22.053 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T20:10:22.269 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-10-05T20:10:22.446 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-10-05T20:10:22.590 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-05T20:10:23.143 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-10-05T20:10:23.333 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:10:23.383 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:10:23.532 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 11/25 2023-10-05T20:10:23.693 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T20:10:23.933 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T20:10:24.135 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 14/25 2023-10-05T20:10:24.298 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T20:10:24.363 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T20:10:24.565 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-05T20:10:24.772 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-10-05T20:10:24.964 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-05T20:10:25.132 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 19/25 2023-10-05T20:10:27.834 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:10:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:10:28.616 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T20:10:28.911 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T20:10:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T20:10:29.164 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:10:47.540 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:10:47.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-10-05T20:10:47.540 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-10-05T20:10:47.541 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:10:49.249 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T20:10:49.310 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T20:10:49.310 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T20:10:49.506 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T20:10:49.506 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:10:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:10:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/25 2023-10-05T20:10:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 6/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 7/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 8/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 9/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 11/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T20:10:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-05T20:10:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/25 2023-10-05T20:10:51.411 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-05T20:10:51.412 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-05T20:10:51.413 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.414 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:10:51.414 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-05T20:10:51.414 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:10:51.414 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:10:51.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2023-10-05T20:10:52.646 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:06:14 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:10:54.225 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 52 M 2023-10-05T20:10:54.226 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 174 k 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-10-05T20:10:54.227 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:10:54.228 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 53 M 2023-10-05T20:10:54.228 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 234 M 2023-10-05T20:10:54.228 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:10:54.992 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): libcephsqlite-17.2.6-1278.g487ffcb5.el8. 301 kB/s | 174 kB 00:00 2023-10-05T20:10:57.726 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-test-17.2.6-1278.g487ffcb5.el8.x86_ 16 MB/s | 52 MB 00:03 2023-10-05T20:15:14.022 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] socat-1.7.4.1-1.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-1.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:15:14.321 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 kB/s | 323 kB 04:16 2023-10-05T20:15:55.415 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] oniguruma-6.8.2-2.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/oniguruma-6.8.2-2.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:15:55.502 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] jq-1.6-7.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/jq-1.6-7.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:15:55.974 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 637 B/s | 187 kB 05:00 2023-10-05T20:15:56.033 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): jq-1.6-7.el8.x86_64.rpm 687 B/s | 202 kB 05:01 2023-10-05T20:20:15.323 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] xmlstarlet-1.6.1-20.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:20:15.820 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 240 B/s | 71 kB 05:01 2023-10-05T20:20:15.820 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:20:15.820 INFO:teuthology.orchestra.run.smithi086.stdout:Total 96 kB/s | 53 MB 09:21 2023-10-05T20:20:15.858 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:20:15.879 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:20:15.879 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:20:16.228 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:20:16.231 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:20:16.735 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:20:16.951 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-10-05T20:20:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-10-05T20:20:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-05T20:20:17.383 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-10-05T20:20:17.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-10-05T20:20:17.721 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T20:20:22.731 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T20:20:22.927 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T20:20:23.204 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T20:20:23.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/6 2023-10-05T20:20:23.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/6 2023-10-05T20:20:23.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-10-05T20:20:23.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-10-05T20:20:23.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-10-05T20:20:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-10-05T20:20:23.840 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-7.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:20:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:20:24.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2023-10-05T20:20:24.895 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:15:46 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:20:26.477 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:20:26.480 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 6.4 k 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 2.3 M 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 1.6 M 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 238 k 2023-10-05T20:20:26.481 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.9 M 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-05T20:20:26.482 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-05T20:20:26.483 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:20:26.484 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2023-10-05T20:20:26.485 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:20:26.485 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 28 M 2023-10-05T20:20:26.485 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 110 M 2023-10-05T20:20:26.486 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:20:28.838 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-17.2.6-1278.g487ffcb5.el8.x86_64.r 47 kB/s | 6.4 kB 00:00 2023-10-05T20:20:28.995 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mgr-17.2.6-1278.g487ffcb5.el8.x86_ 5.3 MB/s | 1.6 MB 00:00 2023-10-05T20:20:29.521 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mon-17.2.6-1278.g487ffcb5.el8.x86_ 5.8 MB/s | 3.9 MB 00:00 2023-10-05T20:20:29.646 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1278.g487f 1.9 MB/s | 238 kB 00:00 2023-10-05T20:20:29.696 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-mds-17.2.6-1278.g487ffcb5.el8.x86_ 2.3 MB/s | 2.3 MB 00:00 2023-10-05T20:20:30.030 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): ceph-osd-17.2.6-1278.g487ffcb5.el8.x86_ 16 MB/s | 17 MB 00:01 2023-10-05T20:24:39.341 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:24:39.535 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 424 B/s | 103 kB 04:09 2023-10-05T20:24:51.557 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-markupsafe-0.23-19.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-markupsafe-0.23-19.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:24:51.558 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-mako-1.0.6-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-mako-1.0.6-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:24:51.651 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 151 B/s | 39 kB 04:21 2023-10-05T20:24:51.751 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-mako-1.0.6-14.el8.noarch.rpm 613 B/s | 157 kB 04:22 2023-10-05T20:24:51.843 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.7 MB/s | 249 kB 00:00 2023-10-05T20:24:51.877 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2023-10-05T20:24:51.919 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 8.3 MB/s | 641 kB 00:00 2023-10-05T20:24:51.944 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-10-05T20:24:52.969 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 41 kB/s | 44 kB 00:01 2023-10-05T20:24:53.386 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 128 kB/s | 185 kB 00:01 2023-10-05T20:24:53.728 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 227 kB/s | 173 kB 00:00 2023-10-05T20:24:53.986 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 641 kB/s | 384 kB 00:00 2023-10-05T20:24:54.012 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 39 kB/s | 11 kB 00:00 2023-10-05T20:24:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 68 kB/s | 18 kB 00:00 2023-10-05T20:24:54.271 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 188 kB/s | 49 kB 00:00 2023-10-05T20:24:54.546 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 195 kB/s | 59 kB 00:00 2023-10-05T20:24:54.604 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 848 kB/s | 283 kB 00:00 2023-10-05T20:24:54.871 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 70 kB/s | 19 kB 00:00 2023-10-05T20:24:55.088 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 29 kB/s | 16 kB 00:00 2023-10-05T20:24:55.113 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 99 kB/s | 24 kB 00:00 2023-10-05T20:24:55.364 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 106 kB/s | 29 kB 00:00 2023-10-05T20:24:55.389 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 100 kB/s | 27 kB 00:00 2023-10-05T20:24:55.714 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 772 kB/s | 251 kB 00:00 2023-10-05T20:24:55.806 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 577 kB/s | 255 kB 00:00 2023-10-05T20:24:55.998 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 301 kB/s | 85 kB 00:00 2023-10-05T20:24:56.048 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 95 kB/s | 23 kB 00:00 2023-10-05T20:28:49.452 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pytz-2017.2-11.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pytz-2017.2-11.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:28:49.686 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 233 B/s | 57 kB 04:10 2023-10-05T20:28:49.686 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:28:49.686 INFO:teuthology.orchestra.run.smithi086.stdout:Total 58 kB/s | 28 MB 08:23 2023-10-05T20:28:49.764 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:28:49.802 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:28:49.802 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:28:50.111 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:28:50.118 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:28:51.191 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:28:51.401 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-05T20:28:51.596 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-05T20:28:51.796 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-10-05T20:28:51.994 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-10-05T20:28:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-10-05T20:28:52.364 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-10-05T20:28:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-10-05T20:28:52.743 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-10-05T20:28:52.970 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-10-05T20:28:53.181 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-10-05T20:28:53.370 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-10-05T20:28:53.576 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-10-05T20:28:53.954 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-10-05T20:28:54.207 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-10-05T20:28:54.407 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-10-05T20:28:54.629 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-10-05T20:28:54.888 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-05T20:28:55.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-10-05T20:28:55.376 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-10-05T20:28:55.556 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-10-05T20:28:55.717 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-05T20:28:55.969 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-05T20:28:56.224 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-05T20:28:56.448 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-10-05T20:28:56.750 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-10-05T20:28:57.059 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-05T20:28:57.397 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 27/32 2023-10-05T20:28:57.567 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/32 2023-10-05T20:28:59.041 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/32 2023-10-05T20:28:59.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 29/32 2023-10-05T20:28:59.592 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 29/32 2023-10-05T20:28:59.762 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 30/32 2023-10-05T20:28:59.978 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 30/32 2023-10-05T20:29:00.145 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 31/32 2023-10-05T20:29:00.162 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 31/32 2023-10-05T20:29:00.270 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 32/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 32/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 6/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-10-05T20:29:00.665 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-10-05T20:29:00.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-10-05T20:29:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-10-05T20:29:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-10-05T20:29:00.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-10-05T20:29:00.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-10-05T20:29:01.797 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-10-05T20:29:01.797 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-10-05T20:29:01.798 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-05T20:29:01.799 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-05T20:29:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-05T20:29:01.801 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-05T20:29:01.801 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:01.801 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:02.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2023-10-05T20:29:02.899 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:24 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:04.296 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:29:04.482 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:04.483 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:29:04.484 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:04.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2023-10-05T20:29:05.444 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:27 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:07.016 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:07.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:07.016 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:29:07.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:07.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 80 k 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:29:07.017 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:07.018 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 80 k 2023-10-05T20:29:07.018 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 364 k 2023-10-05T20:29:07.018 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:29:07.184 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.6-1278.g487ffcb5.el8.noarch.rpm 481 kB/s | 80 kB 00:00 2023-10-05T20:29:07.185 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:29:07.185 INFO:teuthology.orchestra.run.smithi086.stdout:Total 477 kB/s | 80 kB 00:00 2023-10-05T20:29:07.186 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:29:07.193 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:29:07.193 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:29:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:29:07.198 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:29:07.243 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:29:07.666 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:29:07.811 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:29:08.034 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:29:08.642 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:29:08.642 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:08.642 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:29:08.643 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:29:08.643 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:08.643 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:08.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2023-10-05T20:29:09.739 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:31 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:11.301 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:11.301 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:11.301 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-10-05T20:29:11.301 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 162 k 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:29:11.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:11.303 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 162 k 2023-10-05T20:29:11.303 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 490 k 2023-10-05T20:29:11.303 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:29:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.6-1278.g487ffc 872 kB/s | 162 kB 00:00 2023-10-05T20:29:11.489 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:29:11.489 INFO:teuthology.orchestra.run.smithi086.stdout:Total 865 kB/s | 162 kB 00:00 2023-10-05T20:29:11.489 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:29:11.500 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:29:11.500 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:29:11.546 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:29:11.549 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:29:11.656 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:29:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:29:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:29:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T20:29:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:12.886 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:29:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:29:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:12.887 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:13.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2023-10-05T20:29:13.956 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:35 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:15.359 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:29:15.553 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:15.554 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:29:15.555 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:15.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2023-10-05T20:29:16.517 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:38 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:18.119 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 1.7 M 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 23 k 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 14 k 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-05T20:29:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2023-10-05T20:29:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 8.0 M 2023-10-05T20:29:18.122 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:29:18.458 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1278.g487 160 kB/s | 23 kB 00:00 2023-10-05T20:29:18.508 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 852 kB/s | 43 kB 00:00 2023-10-05T20:29:18.525 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): ceph-prometheus-alerts-17.2.6-1278.g487f 69 kB/s | 14 kB 00:00 2023-10-05T20:29:18.601 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): ceph-mgr-dashboard-17.2.6-1278.g487ffcb5 5.9 MB/s | 1.7 MB 00:00 2023-10-05T20:29:18.785 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 122 kB/s | 34 kB 00:00 2023-10-05T20:29:18.918 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 501 kB/s | 196 kB 00:00 2023-10-05T20:29:18.921 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:29:18.921 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-10-05T20:29:18.931 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:29:18.940 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:29:18.940 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:29:19.017 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:29:19.020 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:29:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:29:19.448 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-10-05T20:29:19.644 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-10-05T20:29:19.821 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-10-05T20:29:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 4/6 2023-10-05T20:29:20.237 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 5/6 2023-10-05T20:29:20.721 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T20:29:21.010 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T20:29:21.010 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 1/6 2023-10-05T20:29:21.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 2/6 2023-10-05T20:29:21.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 3/6 2023-10-05T20:29:21.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-10-05T20:29:21.011 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:21.624 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:29:21.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-05T20:29:22.622 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:24:44 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:29:24.200 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 7.4 M 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-10-05T20:29:24.202 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================================ 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2023-10-05T20:29:24.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:29:24.204 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 35 M 2023-10-05T20:29:24.204 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 210 M 2023-10-05T20:29:24.204 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:29:24.957 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-127 14 MB/s | 7.4 MB 00:00 2023-10-05T20:34:24.958 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] openblas-0.3.15-4.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-0.3.15-4.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:34:25.429 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-numpy-1.14.3-10.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-1.14.3-10.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:34:25.510 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] openblas-threads-0.3.15-4.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-threads-0.3.15-4.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:34:26.448 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 16 kB/s | 4.7 MB 05:02 2023-10-05T20:34:26.515 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 kB/s | 3.7 MB 05:01 2023-10-05T20:34:26.631 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 16 kB/s | 4.9 MB 05:02 2023-10-05T20:34:26.657 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-10-05T20:34:26.724 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 2.5 MB/s | 172 kB 00:00 2023-10-05T20:34:26.749 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libgfortran-8.5.0-20.el8.x86_64.rpm 5.4 MB/s | 645 kB 00:00 2023-10-05T20:34:27.233 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 MB/s | 14 MB 00:00 2023-10-05T20:34:27.233 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:34:27.233 INFO:teuthology.orchestra.run.smithi086.stdout:Total 120 kB/s | 35 MB 05:03 2023-10-05T20:34:27.459 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:34:27.477 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:34:27.477 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:34:27.850 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:34:27.853 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:34:28.393 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:34:28.552 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-10-05T20:34:28.680 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-10-05T20:34:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-10-05T20:34:29.414 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-10-05T20:34:29.594 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:34:30.208 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:34:30.335 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-05T20:34:31.033 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-10-05T20:34:31.380 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-05T20:34:31.619 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-05T20:34:33.375 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-05T20:34:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-10-05T20:34:34.827 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T20:34:35.141 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T20:34:35.141 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 1/8 2023-10-05T20:34:35.141 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-10-05T20:34:35.141 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-10-05T20:34:35.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-10-05T20:34:35.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-10-05T20:34:35.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-10-05T20:34:35.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-10-05T20:34:35.786 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-4.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-10-05T20:34:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:34:35.788 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:34:35.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2023-10-05T20:34:36.771 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:29:58 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:34:38.343 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 47 k 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-05T20:34:38.345 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:34:38.346 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 25 M 2023-10-05T20:34:38.347 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:34:38.930 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): ceph-mgr-rook-17.2.6-1278.g487ffcb5.el8 155 kB/s | 47 kB 00:00 2023-10-05T20:38:47.241 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pyasn1-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:38:47.561 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 520 B/s | 126 kB 04:08 2023-10-05T20:38:59.430 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jsonpatch-1.21-2.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-2.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:38:59.430 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jsonpointer-1.10-11.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-1.10-11.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:38:59.593 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 104 B/s | 27 kB 04:20 2023-10-05T20:38:59.618 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 78 B/s | 20 kB 04:20 2023-10-05T20:38:59.677 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 731 kB/s | 43 kB 00:00 2023-10-05T20:38:59.702 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-10-05T20:38:59.978 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 110 kB/s | 33 kB 00:00 2023-10-05T20:39:00.011 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 53 kB/s | 16 kB 00:00 2023-10-05T20:39:00.170 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 424 kB/s | 81 kB 00:00 2023-10-05T20:39:00.270 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 643 kB/s | 64 kB 00:00 2023-10-05T20:39:00.378 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2023-10-05T20:39:00.421 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 2.8 MB/s | 1.2 MB 00:00 2023-10-05T20:43:09.301 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:43:09.605 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 430 B/s | 110 kB 04:22 2023-10-05T20:43:09.606 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:43:09.606 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.9 kB/s | 1.9 MB 08:31 2023-10-05T20:43:09.659 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:43:09.676 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:43:09.676 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:43:09.762 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:43:09.765 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:43:10.240 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:43:10.487 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-05T20:43:10.659 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-10-05T20:43:10.870 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-10-05T20:43:11.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-10-05T20:43:11.265 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-10-05T20:43:11.471 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-10-05T20:43:11.745 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-10-05T20:43:11.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-10-05T20:43:12.979 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-10-05T20:43:13.241 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-10-05T20:43:13.433 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-05T20:43:13.617 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-10-05T20:43:13.767 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 1/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-10-05T20:43:14.059 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-10-05T20:43:14.060 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-05T20:43:14.747 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-05T20:43:14.748 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-05T20:43:14.749 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:43:14.749 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:43:14.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2023-10-05T20:43:15.724 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:38:37 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:43:17.301 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:43:17.302 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:43:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 121 k 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout:Install 5 Packages 2023-10-05T20:43:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:43:17.304 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.0 M 2023-10-05T20:43:17.304 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2023-10-05T20:43:17.304 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:43:17.796 INFO:teuthology.orchestra.run.smithi086.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1278.g487ffcb5.e 299 kB/s | 121 kB 00:00 2023-10-05T20:43:17.905 INFO:teuthology.orchestra.run.smithi086.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 4.3 MB/s | 476 kB 00:00 2023-10-05T20:43:17.955 INFO:teuthology.orchestra.run.smithi086.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.2 MB/s | 60 kB 00:00 2023-10-05T20:47:23.245 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-babel-2.5.1-7.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-babel-2.5.1-7.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:47:23.246 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-jinja2-2.10.1-3.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.10.1-3.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-10-05T20:47:23.602 INFO:teuthology.orchestra.run.smithi086.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.2 kB/s | 538 kB 04:06 2023-10-05T20:47:23.752 INFO:teuthology.orchestra.run.smithi086.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 20 kB/s | 4.8 MB 04:06 2023-10-05T20:47:23.753 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:47:23.753 INFO:teuthology.orchestra.run.smithi086.stdout:Total 25 kB/s | 6.0 MB 04:06 2023-10-05T20:47:23.805 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:47:23.816 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:47:23.816 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:47:23.920 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:47:23.924 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:47:24.239 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:47:24.566 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-10-05T20:47:25.496 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-05T20:47:25.839 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-10-05T20:47:26.095 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-05T20:47:26.268 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T20:47:26.553 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T20:47:26.553 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/5 2023-10-05T20:47:26.553 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-10-05T20:47:26.553 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-10-05T20:47:26.553 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-05T20:47:27.185 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:27.186 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:27.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2023-10-05T20:47:28.137 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:42:50 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:29.734 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 876 k 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:47:29.735 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:29.736 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2023-10-05T20:47:29.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:29.736 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 959 k 2023-10-05T20:47:29.736 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.0 M 2023-10-05T20:47:29.736 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:47:30.068 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 575 kB/s | 83 kB 00:00 2023-10-05T20:47:30.385 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-17.2.6-1278.g487ffcb5.el8.x86_ 1.9 MB/s | 876 kB 00:00 2023-10-05T20:47:30.385 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:47:30.386 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.4 MB/s | 959 kB 00:00 2023-10-05T20:47:30.396 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:47:30.416 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:47:30.417 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:47:30.501 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:47:30.504 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:47:30.672 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:47:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-10-05T20:47:31.125 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T20:47:31.523 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T20:47:31.524 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/2 2023-10-05T20:47:32.151 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-10-05T20:47:32.152 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:32.152 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:47:32.152 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-10-05T20:47:32.152 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:32.152 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:32.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2023-10-05T20:47:33.214 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:42:55 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:34.810 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 267 k 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:47:34.811 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:34.812 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 267 k 2023-10-05T20:47:34.812 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.3 M 2023-10-05T20:47:34.812 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:47:35.356 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-volume-17.2.6-1278.g487ffcb5.el8.noarch.rp 491 kB/s | 267 kB 00:00 2023-10-05T20:47:35.357 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:47:35.357 INFO:teuthology.orchestra.run.smithi086.stdout:Total 490 kB/s | 267 kB 00:00 2023-10-05T20:47:35.357 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:47:35.367 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:47:35.367 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:47:35.401 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:47:35.404 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:47:35.585 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:47:35.792 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:47:36.207 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:47:36.857 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T20:47:36.857 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:36.857 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:47:36.858 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T20:47:36.858 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:36.858 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:37.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2023-10-05T20:47:37.915 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:42:59 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:47:39.499 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 130 k 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 130 k 2023-10-05T20:47:39.500 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 464 k 2023-10-05T20:47:39.501 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:47:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.6-1278.g487ffcb5.el8.x86_64 432 kB/s | 130 kB 00:00 2023-10-05T20:47:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:47:39.801 INFO:teuthology.orchestra.run.smithi086.stdout:Total 430 kB/s | 130 kB 00:00 2023-10-05T20:47:39.802 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:47:39.807 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:47:39.807 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:47:39.855 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:47:39.858 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:47:39.968 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:47:40.207 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:40.500 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:41.113 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:41.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2023-10-05T20:47:42.207 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:04 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:43.604 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:47:43.791 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:43.792 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:47:43.792 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:43.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2023-10-05T20:47:44.732 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:06 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:46.298 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:46.298 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:46.298 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-10-05T20:47:46.298 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 25 k 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:47:46.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:46.300 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2023-10-05T20:47:46.300 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 109 k 2023-10-05T20:47:46.300 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:47:46.537 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.6-1278.g487ffcb5.el8.x86_6 105 kB/s | 25 kB 00:00 2023-10-05T20:47:46.537 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:47:46.537 INFO:teuthology.orchestra.run.smithi086.stdout:Total 104 kB/s | 25 kB 00:00 2023-10-05T20:47:46.538 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:47:46.541 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:47:46.541 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:47:46.543 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:47:46.546 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:47:46.593 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:47:46.812 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:47.070 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:47:47.638 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:47.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2023-10-05T20:47:48.597 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:10 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:49.990 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:47:50.181 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:50.182 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:47:50.182 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:50.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2023-10-05T20:47:51.137 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:13 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:52.545 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:47:52.735 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:52.736 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:47:52.736 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:52.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2023-10-05T20:47:53.680 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:15 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:55.084 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:47:55.274 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:55.275 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:47:55.275 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:55.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2023-10-05T20:47:56.223 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:18 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:47:57.619 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:47:57.816 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:47:57.817 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:47:57.817 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:47:57.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2023-10-05T20:47:58.764 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:20 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:48:00.184 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:48:00.373 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:48:00.374 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:48:00.374 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:48:00.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2023-10-05T20:48:01.348 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:23 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:48:02.751 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T20:48:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:48:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-10-05T20:48:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:48:03.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2023-10-05T20:48:03.902 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:25 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:48:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:48:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:05.485 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-10-05T20:48:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:05.485 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 90 k 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:05.486 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 90 k 2023-10-05T20:48:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 264 k 2023-10-05T20:48:05.487 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:48:06.042 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 163 kB/s | 90 kB 00:00 2023-10-05T20:48:06.043 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:48:06.043 INFO:teuthology.orchestra.run.smithi086.stdout:Total 162 kB/s | 90 kB 00:00 2023-10-05T20:48:06.043 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:48:06.052 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:48:06.053 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:48:06.104 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:48:06.107 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:48:06.219 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:48:06.403 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:07.310 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:48:07.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2023-10-05T20:48:08.370 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:30 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:48:09.958 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:48:09.958 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:09.958 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:48:09.959 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:09.960 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.5 M 2023-10-05T20:48:09.960 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 16 M 2023-10-05T20:48:09.960 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:48:10.319 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 9.6 MB/s | 3.5 MB 00:00 2023-10-05T20:48:10.320 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:48:10.320 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.6 MB/s | 3.5 MB 00:00 2023-10-05T20:48:10.321 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:48:10.332 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:48:10.332 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:48:10.395 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:48:10.397 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:48:10.851 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:48:11.021 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:11.441 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:12.049 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:12.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:12.049 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:48:12.049 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:48:12.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:12.050 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:48:12.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2023-10-05T20:48:13.123 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:43:35 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:48:14.727 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 173 k 2023-10-05T20:48:14.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 173 k 2023-10-05T20:48:14.729 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 552 k 2023-10-05T20:48:14.730 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:48:15.005 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 628 kB/s | 173 kB 00:00 2023-10-05T20:48:15.005 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:48:15.006 INFO:teuthology.orchestra.run.smithi086.stdout:Total 625 kB/s | 173 kB 00:00 2023-10-05T20:48:15.006 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:48:15.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:48:15.016 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:48:15.066 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:48:15.069 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:48:15.185 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:48:15.369 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:15.650 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:16.245 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:48:16.465 DEBUG:teuthology.parallel:result is None 2023-10-05T20:48:16.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T20:48:16.608 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-05T20:48:16.631 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.6-1278.g487ffcb5.el8 2023-10-05T20:48:16.631 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278.g487ffcb5.el8 2023-10-05T20:48:16.631 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278.g487ffcb5 is installed. 2023-10-05T20:48:16.632 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T20:48:16.632 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:16.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T20:48:16.700 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T20:48:16.700 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:16.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T20:48:16.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T20:48:16.833 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T20:48:16.834 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:16.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T20:48:16.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T20:48:16.967 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T20:48:16.967 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:16.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T20:48:17.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T20:48:17.101 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T20:48:17.165 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T20:48:17.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2023-10-05T20:48:17.192 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T20:48:17.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T20:48:17.259 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T20:48:17.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T20:48:17.335 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T20:48:17.335 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T20:48:17.335 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411757', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqsJPXSyQ8/UoKTpMll+bKPWbkWz3arfKZaC6WX3RqXyZCaa/eZYDhqtXH65cBInWzNcuT84sUvnATNTf/gbwA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T20:48:17.336 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T20:48:17.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T20:48:17.453 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:17.453 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T20:48:17.511 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T20:48:17.511 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2023-10-05T20:48:17.568 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 390760 Links: 1 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-10-05 20:48:11.195437117 +0000 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-10-05 19:49:43.370311200 +0000 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-10-05 19:49:43.370311200 +0000 2023-10-05T20:48:17.569 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-10-05T20:48:17.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T20:48:17.634 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-10-05T20:48:17.634 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-10-05T20:48:17.634 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000242126 s, 2.1 MB/s 2023-10-05T20:48:17.635 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T20:48:17.694 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 379829 Links: 1 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-10-05 20:48:11.195437117 +0000 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-10-05 19:49:43.877300543 +0000 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-10-05 19:49:43.877300543 +0000 2023-10-05T20:48:17.753 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-10-05T20:48:17.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T20:48:17.818 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-10-05T20:48:17.818 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-10-05T20:48:17.819 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000194231 s, 2.6 MB/s 2023-10-05T20:48:17.819 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T20:48:17.878 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2023-10-05T20:48:17.936 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 379880 Links: 1 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-10-05 20:48:11.195437117 +0000 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-10-05 19:49:44.388289803 +0000 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-10-05 19:49:44.388289803 +0000 2023-10-05T20:48:17.937 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-10-05T20:48:17.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T20:48:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-10-05T20:48:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-10-05T20:48:18.003 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000211745 s, 2.4 MB/s 2023-10-05T20:48:18.003 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T20:48:18.062 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2023-10-05T20:48:18.120 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 397790 Links: 1 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-10-05 20:48:11.195437117 +0000 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-10-05 19:49:44.883279399 +0000 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-10-05 19:49:44.883279399 +0000 2023-10-05T20:48:18.121 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-10-05T20:48:18.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T20:48:18.186 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-10-05T20:48:18.186 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-10-05T20:48:18.186 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000220157 s, 2.3 MB/s 2023-10-05T20:48:18.187 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T20:48:18.246 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T20:48:18.247 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T20:48:18.247 INFO:tasks.ceph:Generating config... 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T20:48:18.248 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T20:48:18.248 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T20:48:18.248 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T20:48:18.248 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T20:48:18.249 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T20:48:18.249 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T20:48:18.249 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T20:48:18.249 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T20:48:18.249 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-05T20:48:18.249 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T20:48:18.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T20:48:18.341 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T20:48:18.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T20:48:18.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T20:48:18.451 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.86')] 2023-10-05T20:48:18.452 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.86'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption_keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T20:48:18.452 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:18.453 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T20:48:18.509 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.86 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:18.588 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:48:18.588 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid d60c04eb-231a-4399-a9de-74389c06e19a 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = octopus 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:fsid d60c04eb-231a-4399-a9de-74389c06e19a 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2023-10-05T20:48:18.588036+0000 2023-10-05T20:48:18.589 INFO:teuthology.orchestra.run.smithi086.stdout:created 2023-10-05T20:48:18.588036+0000 2023-10-05T20:48:18.590 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 15 (octopus) 2023-10-05T20:48:18.590 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2023-10-05T20:48:18.590 INFO:teuthology.orchestra.run.smithi086.stdout:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-10-05T20:48:18.590 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T20:48:18.591 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T20:48:18.648 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d60c04eb-231a-4399-a9de-74389c06e19a... 2023-10-05T20:48:18.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T20:48:18.733 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2023-10-05T20:48:18.733 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2023-10-05T20:48:18.733 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2023-10-05T20:48:18.734 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2023-10-05T20:48:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T20:48:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = 172.21.15.86 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = d60c04eb-231a-4399-a9de-74389c06e19a 2023-10-05T20:48:18.737 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2023-10-05T20:48:18.738 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2023-10-05T20:48:18.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2023-10-05T20:48:18.740 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: osd shutdown pgref assert = True 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-10-05T20:48:18.741 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2023-10-05T20:48:18.742 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: debug rgw = 20 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt require ssl = False 2023-10-05T20:48:18.744 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms backend = testing 2023-10-05T20:48:18.745 INFO:teuthology.orchestra.run.smithi086.stdout: rgw crypt s3 kms encryption_keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-10-05T20:48:18.745 INFO:teuthology.orchestra.run.smithi086.stdout: setgroup = ceph 2023-10-05T20:48:18.745 INFO:teuthology.orchestra.run.smithi086.stdout: setuser = ceph 2023-10-05T20:48:18.745 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2023-10-05T20:48:18.746 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T20:48:18.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T20:48:18.839 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T20:48:18.840 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:18.840 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T20:48:18.857 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:18.857 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T20:48:18.915 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:48:18.915 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:18.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T20:48:18.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T20:48:18.992 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:18.992 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:19.050 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T20:48:19.051 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T20:48:19.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T20:48:19.141 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T20:48:19.143 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T20:48:19.143 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T20:48:19.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T20:48:19.183 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T20:48:19.195 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T20:48:19.195 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T20:48:19.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T20:48:19.263 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T20:48:19.263 INFO:tasks.ceph:role: osd.0 2023-10-05T20:48:19.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:48:19.263 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T20:48:19.337 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T20:48:19.337 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T20:48:19.337 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T20:48:19.337 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T20:48:19.338 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T20:48:19.771 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-10-05T20:48:19.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-10-05T20:48:19.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T20:48:19.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T20:48:19.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T20:48:20.049 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T20:48:20.050 INFO:tasks.ceph:role: osd.1 2023-10-05T20:48:20.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:48:20.050 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T20:48:20.124 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T20:48:20.125 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T20:48:20.125 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T20:48:20.550 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-10-05T20:48:20.552 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-10-05T20:48:20.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-05T20:48:20.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T20:48:20.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T20:48:20.756 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T20:48:20.757 INFO:tasks.ceph:role: osd.2 2023-10-05T20:48:20.757 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:48:20.757 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T20:48:20.831 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T20:48:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T20:48:20.831 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T20:48:20.832 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T20:48:20.833 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T20:48:21.284 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-10-05T20:48:21.286 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-10-05T20:48:21.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-05T20:48:21.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T20:48:21.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:21.579 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.577+0000 7fc2f1e97540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T20:48:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.579+0000 7fc2f1e97540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T20:48:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.579+0000 7fc2f1e97540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T20:48:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.579+0000 7fc2f1e97540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T20:48:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.579+0000 7fc2f1e97540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T20:48:21.582 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:21.580+0000 7fc2f1e97540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T20:48:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:22.108+0000 7fc2f1e97540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:22.871 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:22.869+0000 7fc2f1e97540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:23.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T20:48:23.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T20:48:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:23.783+0000 7f4614b78540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T20:48:24.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:24.306+0000 7f4614b78540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:25.070 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.068+0000 7f4614b78540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:25.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T20:48:25.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.975+0000 7f8d06f15540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.975+0000 7f8d06f15540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.975+0000 7f8d06f15540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.975+0000 7f8d06f15540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.975+0000 7f8d06f15540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T20:48:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:25.976+0000 7f8d06f15540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T20:48:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:26.499+0000 7f8d06f15540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:27.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:48:27.266+0000 7f8d06f15540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T20:48:28.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T20:48:28.083 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T20:48:28.084 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:28.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T20:48:28.152 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:28.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T20:48:28.221 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:28.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T20:48:28.290 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:28.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T20:48:28.357 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:28.357 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T20:48:28.415 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T20:48:28.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T20:48:28.480 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQATIR9lOklcCBAA8vfh7AD3xR5wXO1sFTrm2A== 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAVIR9lY8V/IhAAbUzzbM0MI2CPMNuyMshmCw== 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAXIR9l+Vq8LhAAnQsMd1+Ld+G2GvNK2gVPsQ== 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAZIR9l5isyOhAAV5E8/2706iCLrYI7JafZgA== 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2023-10-05T20:48:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQATIR9l1OHpChAAbFUlUrmZpXnNE2dv52AQQw== 2023-10-05T20:48:28.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T20:48:28.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T20:48:28.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T20:48:28.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T20:48:28.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T20:48:28.774 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T20:48:28.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T20:48:28.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T20:48:28.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T20:48:28.926 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T20:48:28.984 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T20:48:28.985 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T20:48:28.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T20:48:29.027 INFO:tasks.ceph.mon.a:Started 2023-10-05T20:48:29.027 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T20:48:29.027 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T20:48:29.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T20:48:29.029 INFO:tasks.ceph.mgr.x:Started 2023-10-05T20:48:29.029 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T20:48:29.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T20:48:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2023-10-05T20:48:29.243 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T20:48:29.243 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:29.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T20:48:29.271 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:29.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T20:48:29.338 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:48:29.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T20:48:29.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 6e7917e6-a209-45f7-b03c-855e6ddb30b8 0 2023-10-05T20:48:29.428 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:29.426+0000 7f1873b721c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T20:48:29.602 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:29.600+0000 7f1873b721c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-05T20:48:29.680 INFO:teuthology.orchestra.run.smithi086.stdout:0 2023-10-05T20:48:29.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7c3965a7-b09a-4fe4-9205-bc4c0a5028db 1 2023-10-05T20:48:29.898 INFO:teuthology.orchestra.run.smithi086.stdout:1 2023-10-05T20:48:29.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 1171759f-5db6-4d73-bf0e-d66423f5cbc1 2 2023-10-05T20:48:29.950 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:29.948+0000 7f1873b721c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T20:48:30.121 INFO:teuthology.orchestra.run.smithi086.stdout:2 2023-10-05T20:48:30.133 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T20:48:30.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T20:48:30.175 INFO:tasks.ceph.osd.0:Started 2023-10-05T20:48:30.175 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T20:48:30.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T20:48:30.177 INFO:tasks.ceph.osd.1:Started 2023-10-05T20:48:30.177 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T20:48:30.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T20:48:30.180 INFO:tasks.ceph.osd.2:Started 2023-10-05T20:48:30.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T20:48:30.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:30.384 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":5,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:48:30.118607+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:48:30.396 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T20:48:30.396 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T20:48:30.508 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:30.508+0000 7f1873b721c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-05T20:48:30.646 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:30.645+0000 7f1873b721c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-05T20:48:30.699 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T20:48:30.784 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-10-05T20:48:30.783+0000 7f702dcc0540 -1 Falling back to public interface 2023-10-05T20:48:30.785 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-10-05T20:48:30.784+0000 7fe75406a540 -1 Falling back to public interface 2023-10-05T20:48:30.796 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-10-05T20:48:30.796+0000 7f2875cd4540 -1 Falling back to public interface 2023-10-05T20:48:30.888 INFO:teuthology.misc.health.smithi086.stdout: 2023-10-05T20:48:30.888 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":5,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:48:30.118607+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:48:30.899 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-05T20:48:30.990 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:30.989+0000 7f1873b721c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T20:48:31.701 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:31.700+0000 7f1873b721c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T20:48:31.771 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:31.770+0000 7f1873b721c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T20:48:31.903 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:31.901+0000 7f1873b721c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T20:48:31.972 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:31.970+0000 7f1873b721c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T20:48:32.148 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.146+0000 7f1873b721c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T20:48:32.226 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.224+0000 7f1873b721c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T20:48:32.301 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.299+0000 7f1873b721c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T20:48:32.388 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.386+0000 7f1873b721c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T20:48:32.564 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.563+0000 7f1873b721c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T20:48:32.642 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.641+0000 7f1873b721c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T20:48:32.924 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.922+0000 7f1873b721c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T20:48:32.992 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:32.990+0000 7f1873b721c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T20:48:33.501 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:33.499+0000 7f1873b721c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T20:48:33.601 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-10-05T20:48:33.599+0000 7f702dcc0540 -1 osd.0 0 log_to_monitors true 2023-10-05T20:48:33.606 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-10-05T20:48:33.604+0000 7fe75406a540 -1 osd.1 0 log_to_monitors true 2023-10-05T20:48:33.620 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-10-05T20:48:33.618+0000 7f2875cd4540 -1 osd.2 0 log_to_monitors true 2023-10-05T20:48:33.823 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:33.821+0000 7f1873b721c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T20:48:33.960 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:33.958+0000 7f1873b721c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T20:48:34.306 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:34.305+0000 7f1873b721c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T20:48:34.447 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:34.445+0000 7f1873b721c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T20:48:34.592 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:34.590+0000 7f1873b721c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T20:48:34.714 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:34.710+0000 7f1873b721c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T20:48:35.825 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-10-05T20:48:35.823+0000 7f1838253700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T20:48:35.875 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-10-05T20:48:35.873+0000 7f70248c5700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T20:48:35.877 INFO:tasks.ceph.osd.0.smithi086.stderr:2023-10-05T20:48:35.875+0000 7f701d6bc700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T20:48:35.912 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-10-05T20:48:35.910+0000 7f286c8d9700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T20:48:35.913 INFO:tasks.ceph.osd.2.smithi086.stderr:2023-10-05T20:48:35.912+0000 7f28656d0700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T20:48:35.925 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-10-05T20:48:35.923+0000 7fe74ac6f700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T20:48:35.927 INFO:tasks.ceph.osd.1.smithi086.stderr:2023-10-05T20:48:35.925+0000 7fe743a66700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T20:48:37.201 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T20:48:37.513 INFO:teuthology.misc.health.smithi086.stdout: 2023-10-05T20:48:37.514 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":8,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:48:36.875323+0000","last_up_change":"2023-10-05T20:48:36.875323+0000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":114418},{"type":"v1","addr":"172.21.15.86:6801","nonce":114418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":114418},{"type":"v1","addr":"172.21.15.86:6803","nonce":114418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":114418},{"type":"v1","addr":"172.21.15.86:6807","nonce":114418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":114418},{"type":"v1","addr":"172.21.15.86:6805","nonce":114418}]},"public_addr":"172.21.15.86:6801/114418","cluster_addr":"172.21.15.86:6803/114418","heartbeat_back_addr":"172.21.15.86:6807/114418","heartbeat_front_addr":"172.21.15.86:6805/114418","state":["exists","up"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":114419},{"type":"v1","addr":"172.21.15.86:6809","nonce":114419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":114419},{"type":"v1","addr":"172.21.15.86:6811","nonce":114419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":114419},{"type":"v1","addr":"172.21.15.86:6815","nonce":114419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":114419},{"type":"v1","addr":"172.21.15.86:6813","nonce":114419}]},"public_addr":"172.21.15.86:6809/114419","cluster_addr":"172.21.15.86:6811/114419","heartbeat_back_addr":"172.21.15.86:6815/114419","heartbeat_front_addr":"172.21.15.86:6813/114419","state":["exists","up"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":114448},{"type":"v1","addr":"172.21.15.86:6817","nonce":114448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":114448},{"type":"v1","addr":"172.21.15.86:6819","nonce":114448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":114448},{"type":"v1","addr":"172.21.15.86:6823","nonce":114448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":114448},{"type":"v1","addr":"172.21.15.86:6821","nonce":114448}]},"public_addr":"172.21.15.86:6817/114448","cluster_addr":"172.21.15.86:6819/114448","heartbeat_back_addr":"172.21.15.86:6823/114448","heartbeat_front_addr":"172.21.15.86:6821/114448","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:48:37.526 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-05T20:48:37.526 INFO:tasks.ceph:Creating RBD pool 2023-10-05T20:48:37.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T20:48:37.932 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2023-10-05T20:48:37.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T20:48:38.937 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T20:48:38.952 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T20:48:38.952 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T20:48:38.953 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T20:48:38.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T20:48:38.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T20:48:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:39.333 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":114006},{"type":"v1","addr":"172.21.15.86:6825","nonce":114006}]},"active_addr":"172.21.15.86:6825/114006","active_change":"2023-10-05T20:48:34.725024+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3019054127}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":4272651176}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3333253909}]}]} 2023-10-05T20:48:39.336 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T20:48:39.336 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T20:48:39.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T20:48:39.645 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:39.645 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":10,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:48:38.927286+0000","last_up_change":"2023-10-05T20:48:36.875323+0000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-05T20:48:37.849330+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-05T20:48:38.858982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":114418},{"type":"v1","addr":"172.21.15.86:6801","nonce":114418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":114418},{"type":"v1","addr":"172.21.15.86:6803","nonce":114418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":114418},{"type":"v1","addr":"172.21.15.86:6807","nonce":114418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":114418},{"type":"v1","addr":"172.21.15.86:6805","nonce":114418}]},"public_addr":"172.21.15.86:6801/114418","cluster_addr":"172.21.15.86:6803/114418","heartbeat_back_addr":"172.21.15.86:6807/114418","heartbeat_front_addr":"172.21.15.86:6805/114418","state":["exists","up"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":114419},{"type":"v1","addr":"172.21.15.86:6809","nonce":114419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":114419},{"type":"v1","addr":"172.21.15.86:6811","nonce":114419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":114419},{"type":"v1","addr":"172.21.15.86:6815","nonce":114419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":114419},{"type":"v1","addr":"172.21.15.86:6813","nonce":114419}]},"public_addr":"172.21.15.86:6809/114419","cluster_addr":"172.21.15.86:6811/114419","heartbeat_back_addr":"172.21.15.86:6815/114419","heartbeat_front_addr":"172.21.15.86:6813/114419","state":["exists","up"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":114448},{"type":"v1","addr":"172.21.15.86:6817","nonce":114448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":114448},{"type":"v1","addr":"172.21.15.86:6819","nonce":114448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":114448},{"type":"v1","addr":"172.21.15.86:6823","nonce":114448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":114448},{"type":"v1","addr":"172.21.15.86:6821","nonce":114448}]},"public_addr":"172.21.15.86:6817/114448","cluster_addr":"172.21.15.86:6819/114448","heartbeat_back_addr":"172.21.15.86:6823/114448","heartbeat_front_addr":"172.21.15.86:6821/114448","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.571288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.641704+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.588127+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:48:39.658 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T20:48:39.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T20:48:39.975 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:39.975 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":11,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:48:39.929729+0000","last_up_change":"2023-10-05T20:48:36.875323+0000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-05T20:48:37.849330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-05T20:48:38.858982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":114418},{"type":"v1","addr":"172.21.15.86:6801","nonce":114418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":114418},{"type":"v1","addr":"172.21.15.86:6803","nonce":114418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":114418},{"type":"v1","addr":"172.21.15.86:6807","nonce":114418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":114418},{"type":"v1","addr":"172.21.15.86:6805","nonce":114418}]},"public_addr":"172.21.15.86:6801/114418","cluster_addr":"172.21.15.86:6803/114418","heartbeat_back_addr":"172.21.15.86:6807/114418","heartbeat_front_addr":"172.21.15.86:6805/114418","state":["exists","up"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":114419},{"type":"v1","addr":"172.21.15.86:6809","nonce":114419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":114419},{"type":"v1","addr":"172.21.15.86:6811","nonce":114419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":114419},{"type":"v1","addr":"172.21.15.86:6815","nonce":114419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":114419},{"type":"v1","addr":"172.21.15.86:6813","nonce":114419}]},"public_addr":"172.21.15.86:6809/114419","cluster_addr":"172.21.15.86:6811/114419","heartbeat_back_addr":"172.21.15.86:6815/114419","heartbeat_front_addr":"172.21.15.86:6813/114419","state":["exists","up"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":114448},{"type":"v1","addr":"172.21.15.86:6817","nonce":114448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":114448},{"type":"v1","addr":"172.21.15.86:6819","nonce":114448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":114448},{"type":"v1","addr":"172.21.15.86:6823","nonce":114448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":114448},{"type":"v1","addr":"172.21.15.86:6821","nonce":114448}]},"public_addr":"172.21.15.86:6817/114448","cluster_addr":"172.21.15.86:6819/114448","heartbeat_back_addr":"172.21.15.86:6823/114448","heartbeat_front_addr":"172.21.15.86:6821/114448","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.571288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.641704+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.588127+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:48:39.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T20:48:39.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T20:48:39.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T20:48:40.170 INFO:teuthology.orchestra.run.smithi086.stdout:34359738372 2023-10-05T20:48:40.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T20:48:40.171 INFO:teuthology.orchestra.run.smithi086.stdout:34359738370 2023-10-05T20:48:40.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T20:48:40.241 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-10-05T20:48:40.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T20:48:40.559 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-10-05T20:48:40.561 INFO:teuthology.orchestra.run.smithi086.stdout:34359738369 2023-10-05T20:48:40.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-10-05T20:48:40.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2023-10-05T20:48:40.615 INFO:teuthology.orchestra.run.smithi086.stdout:34359738370 2023-10-05T20:48:40.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-10-05T20:48:41.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T20:48:41.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T20:48:41.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T20:48:41.919 INFO:teuthology.orchestra.run.smithi086.stdout:34359738372 2023-10-05T20:48:41.924 INFO:teuthology.orchestra.run.smithi086.stdout:34359738370 2023-10-05T20:48:41.933 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-10-05T20:48:41.933 DEBUG:teuthology.parallel:result is None 2023-10-05T20:48:41.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.2 2023-10-05T20:48:41.940 DEBUG:teuthology.parallel:result is None 2023-10-05T20:48:41.954 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-10-05T20:48:41.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2023-10-05T20:48:41.968 DEBUG:teuthology.parallel:result is None 2023-10-05T20:48:41.968 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T20:48:41.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T20:48:42.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:42.312 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-10-05T20:48:42.323 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-05T20:48:40.931302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":892120,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313680680,"statfs":{"total":322122547200,"available":321209016320,"internally_reserved":0,"allocated":262144,"data_stored":63644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.112551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.941908+0000","last_change":"2023-10-05T20:48:38.940699+0000","last_active":"2023-10-05T20:48:38.941908+0000","last_peered":"2023-10-05T20:48:38.941908+0000","last_clean":"2023-10-05T20:48:38.941908+0000","last_became_active":"2023-10-05T20:48:38.940439+0000","last_became_peered":"2023-10-05T20:48:38.940439+0000","last_unstale":"2023-10-05T20:48:38.941908+0000","last_undegraded":"2023-10-05T20:48:38.941908+0000","last_fullsized":"2023-10-05T20:48:38.941908+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:52:26.850920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.944577+0000","last_change":"2023-10-05T20:48:38.943623+0000","last_active":"2023-10-05T20:48:38.944577+0000","last_peered":"2023-10-05T20:48:38.944577+0000","last_clean":"2023-10-05T20:48:38.944577+0000","last_became_active":"2023-10-05T20:48:38.943356+0000","last_became_peered":"2023-10-05T20:48:38.943356+0000","last_unstale":"2023-10-05T20:48:38.944577+0000","last_undegraded":"2023-10-05T20:48:38.944577+0000","last_fullsized":"2023-10-05T20:48:38.944577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T20:57:09.214097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.940185+0000","last_change":"2023-10-05T20:48:38.938211+0000","last_active":"2023-10-05T20:48:38.940185+0000","last_peered":"2023-10-05T20:48:38.940185+0000","last_clean":"2023-10-05T20:48:38.940185+0000","last_became_active":"2023-10-05T20:48:38.937928+0000","last_became_peered":"2023-10-05T20:48:38.937928+0000","last_unstale":"2023-10-05T20:48:38.940185+0000","last_undegraded":"2023-10-05T20:48:38.940185+0000","last_fullsized":"2023-10-05T20:48:38.940185+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:04:25.547862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.940288+0000","last_change":"2023-10-05T20:48:38.938386+0000","last_active":"2023-10-05T20:48:38.940288+0000","last_peered":"2023-10-05T20:48:38.940288+0000","last_clean":"2023-10-05T20:48:38.940288+0000","last_became_active":"2023-10-05T20:48:38.937945+0000","last_became_peered":"2023-10-05T20:48:38.937945+0000","last_unstale":"2023-10-05T20:48:38.940288+0000","last_undegraded":"2023-10-05T20:48:38.940288+0000","last_fullsized":"2023-10-05T20:48:38.940288+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:39.661815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.936175+0000","last_change":"2023-10-05T20:48:38.936175+0000","last_active":"2023-10-05T20:48:38.936175+0000","last_peered":"2023-10-05T20:48:38.936175+0000","last_clean":"2023-10-05T20:48:38.936175+0000","last_became_active":"2023-10-05T20:48:38.935910+0000","last_became_peered":"2023-10-05T20:48:38.935910+0000","last_unstale":"2023-10-05T20:48:38.936175+0000","last_undegraded":"2023-10-05T20:48:38.936175+0000","last_fullsized":"2023-10-05T20:48:38.936175+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:50.391391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'85","reported_seq":120,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-05T20:48:40.128535+0000","last_change":"2023-10-05T20:48:39.935388+0000","last_active":"2023-10-05T20:48:40.128535+0000","last_peered":"2023-10-05T20:48:40.128535+0000","last_clean":"2023-10-05T20:48:40.128535+0000","last_became_active":"2023-10-05T20:48:39.934653+0000","last_became_peered":"2023-10-05T20:48:39.934653+0000","last_unstale":"2023-10-05T20:48:40.128535+0000","last_undegraded":"2023-10-05T20:48:40.128535+0000","last_fullsized":"2023-10-05T20:48:40.128535+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:38.927286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:38.927286+0000","last_clean_scrub_stamp":"2023-10-05T20:48:38.927286+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:40.663037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.941910+0000","last_change":"2023-10-05T20:48:38.940796+0000","last_active":"2023-10-05T20:48:38.941910+0000","last_peered":"2023-10-05T20:48:38.941910+0000","last_clean":"2023-10-05T20:48:38.941910+0000","last_became_active":"2023-10-05T20:48:38.940357+0000","last_became_peered":"2023-10-05T20:48:38.940357+0000","last_unstale":"2023-10-05T20:48:38.941910+0000","last_undegraded":"2023-10-05T20:48:38.941910+0000","last_fullsized":"2023-10-05T20:48:38.941910+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:00:33.005180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.942252+0000","last_change":"2023-10-05T20:48:38.941596+0000","last_active":"2023-10-05T20:48:38.942252+0000","last_peered":"2023-10-05T20:48:38.942252+0000","last_clean":"2023-10-05T20:48:38.942252+0000","last_became_active":"2023-10-05T20:48:38.941334+0000","last_became_peered":"2023-10-05T20:48:38.941334+0000","last_unstale":"2023-10-05T20:48:38.942252+0000","last_undegraded":"2023-10-05T20:48:38.942252+0000","last_fullsized":"2023-10-05T20:48:38.942252+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:49:17.030799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.942777+0000","last_change":"2023-10-05T20:48:38.941791+0000","last_active":"2023-10-05T20:48:38.942777+0000","last_peered":"2023-10-05T20:48:38.942777+0000","last_clean":"2023-10-05T20:48:38.942777+0000","last_became_active":"2023-10-05T20:48:38.941508+0000","last_became_peered":"2023-10-05T20:48:38.941508+0000","last_unstale":"2023-10-05T20:48:38.942777+0000","last_undegraded":"2023-10-05T20:48:38.942777+0000","last_fullsized":"2023-10-05T20:48:38.942777+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:20.441297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297372,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560228,"statfs":{"total":107374182400,"available":107069673472,"internally_reserved":0,"allocated":86016,"data_stored":20972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297372,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560228,"statfs":{"total":107374182400,"available":107069673472,"internally_reserved":0,"allocated":86016,"data_stored":20972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297376,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560224,"statfs":{"total":107374182400,"available":107069669376,"internally_reserved":0,"allocated":90112,"data_stored":21700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T20:48:42.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T20:48:42.626 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:42.628 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-10-05T20:48:42.640 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-05T20:48:40.931302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":892120,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313680680,"statfs":{"total":322122547200,"available":321209016320,"internally_reserved":0,"allocated":262144,"data_stored":63644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.112551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.941908+0000","last_change":"2023-10-05T20:48:38.940699+0000","last_active":"2023-10-05T20:48:38.941908+0000","last_peered":"2023-10-05T20:48:38.941908+0000","last_clean":"2023-10-05T20:48:38.941908+0000","last_became_active":"2023-10-05T20:48:38.940439+0000","last_became_peered":"2023-10-05T20:48:38.940439+0000","last_unstale":"2023-10-05T20:48:38.941908+0000","last_undegraded":"2023-10-05T20:48:38.941908+0000","last_fullsized":"2023-10-05T20:48:38.941908+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:52:26.850920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.944577+0000","last_change":"2023-10-05T20:48:38.943623+0000","last_active":"2023-10-05T20:48:38.944577+0000","last_peered":"2023-10-05T20:48:38.944577+0000","last_clean":"2023-10-05T20:48:38.944577+0000","last_became_active":"2023-10-05T20:48:38.943356+0000","last_became_peered":"2023-10-05T20:48:38.943356+0000","last_unstale":"2023-10-05T20:48:38.944577+0000","last_undegraded":"2023-10-05T20:48:38.944577+0000","last_fullsized":"2023-10-05T20:48:38.944577+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T20:57:09.214097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.940185+0000","last_change":"2023-10-05T20:48:38.938211+0000","last_active":"2023-10-05T20:48:38.940185+0000","last_peered":"2023-10-05T20:48:38.940185+0000","last_clean":"2023-10-05T20:48:38.940185+0000","last_became_active":"2023-10-05T20:48:38.937928+0000","last_became_peered":"2023-10-05T20:48:38.937928+0000","last_unstale":"2023-10-05T20:48:38.940185+0000","last_undegraded":"2023-10-05T20:48:38.940185+0000","last_fullsized":"2023-10-05T20:48:38.940185+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:04:25.547862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.940288+0000","last_change":"2023-10-05T20:48:38.938386+0000","last_active":"2023-10-05T20:48:38.940288+0000","last_peered":"2023-10-05T20:48:38.940288+0000","last_clean":"2023-10-05T20:48:38.940288+0000","last_became_active":"2023-10-05T20:48:38.937945+0000","last_became_peered":"2023-10-05T20:48:38.937945+0000","last_unstale":"2023-10-05T20:48:38.940288+0000","last_undegraded":"2023-10-05T20:48:38.940288+0000","last_fullsized":"2023-10-05T20:48:38.940288+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:39.661815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.936175+0000","last_change":"2023-10-05T20:48:38.936175+0000","last_active":"2023-10-05T20:48:38.936175+0000","last_peered":"2023-10-05T20:48:38.936175+0000","last_clean":"2023-10-05T20:48:38.936175+0000","last_became_active":"2023-10-05T20:48:38.935910+0000","last_became_peered":"2023-10-05T20:48:38.935910+0000","last_unstale":"2023-10-05T20:48:38.936175+0000","last_undegraded":"2023-10-05T20:48:38.936175+0000","last_fullsized":"2023-10-05T20:48:38.936175+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:50.391391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'85","reported_seq":120,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-05T20:48:40.128535+0000","last_change":"2023-10-05T20:48:39.935388+0000","last_active":"2023-10-05T20:48:40.128535+0000","last_peered":"2023-10-05T20:48:40.128535+0000","last_clean":"2023-10-05T20:48:40.128535+0000","last_became_active":"2023-10-05T20:48:39.934653+0000","last_became_peered":"2023-10-05T20:48:39.934653+0000","last_unstale":"2023-10-05T20:48:40.128535+0000","last_undegraded":"2023-10-05T20:48:40.128535+0000","last_fullsized":"2023-10-05T20:48:40.128535+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:38.927286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:38.927286+0000","last_clean_scrub_stamp":"2023-10-05T20:48:38.927286+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:40.663037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.941910+0000","last_change":"2023-10-05T20:48:38.940796+0000","last_active":"2023-10-05T20:48:38.941910+0000","last_peered":"2023-10-05T20:48:38.941910+0000","last_clean":"2023-10-05T20:48:38.941910+0000","last_became_active":"2023-10-05T20:48:38.940357+0000","last_became_peered":"2023-10-05T20:48:38.940357+0000","last_unstale":"2023-10-05T20:48:38.941910+0000","last_undegraded":"2023-10-05T20:48:38.941910+0000","last_fullsized":"2023-10-05T20:48:38.941910+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:00:33.005180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.942252+0000","last_change":"2023-10-05T20:48:38.941596+0000","last_active":"2023-10-05T20:48:38.942252+0000","last_peered":"2023-10-05T20:48:38.942252+0000","last_clean":"2023-10-05T20:48:38.942252+0000","last_became_active":"2023-10-05T20:48:38.941334+0000","last_became_peered":"2023-10-05T20:48:38.941334+0000","last_unstale":"2023-10-05T20:48:38.942252+0000","last_undegraded":"2023-10-05T20:48:38.942252+0000","last_fullsized":"2023-10-05T20:48:38.942252+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:49:17.030799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T20:48:38.942777+0000","last_change":"2023-10-05T20:48:38.941791+0000","last_active":"2023-10-05T20:48:38.942777+0000","last_peered":"2023-10-05T20:48:38.942777+0000","last_clean":"2023-10-05T20:48:38.942777+0000","last_became_active":"2023-10-05T20:48:38.941508+0000","last_became_peered":"2023-10-05T20:48:38.941508+0000","last_unstale":"2023-10-05T20:48:38.942777+0000","last_undegraded":"2023-10-05T20:48:38.942777+0000","last_fullsized":"2023-10-05T20:48:38.942777+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T20:48:37.926064+0000","last_clean_scrub_stamp":"2023-10-05T20:48:37.926064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:29:20.441297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297372,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560228,"statfs":{"total":107374182400,"available":107069673472,"internally_reserved":0,"allocated":86016,"data_stored":20972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297372,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560228,"statfs":{"total":107374182400,"available":107069673472,"internally_reserved":0,"allocated":86016,"data_stored":20972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297376,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560224,"statfs":{"total":107374182400,"available":107069669376,"internally_reserved":0,"allocated":90112,"data_stored":21700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T20:48:42.641 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T20:48:42.641 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T20:48:42.641 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T20:48:42.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T20:48:42.990 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:48:42.990 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T20:48:43.004 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T20:48:43.004 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T20:48:43.017 DEBUG:tasks.rgw:config is {'client.0': None} 2023-10-05T20:48:43.017 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T20:48:43.018 INFO:tasks.rgw:Creating data pools 2023-10-05T20:48:43.018 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T20:48:43.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-10-05T20:48:43.961 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T20:48:43.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T20:48:44.983 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T20:48:44.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T20:48:45.972 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T20:48:45.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T20:48:46.988 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T20:48:47.004 DEBUG:tasks.rgw:Pools created 2023-10-05T20:48:47.005 INFO:tasks.rgw:Starting rgw... 2023-10-05T20:48:47.005 INFO:tasks.rgw:rgw client.0 config is {} 2023-10-05T20:48:47.005 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T20:48:47.005 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T20:48:47.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T20:48:47.007 INFO:tasks.rgw.client.0:Started 2023-10-05T20:48:47.007 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:48:47.007 DEBUG:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:48:47.039 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T20:48:47.039 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2023-10-05T20:48:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T20:48:47.040 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 80: Connection refused 2023-10-05T20:48:48.042 DEBUG:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:48:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T20:48:48.070 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T20:48:48.070 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2023-10-05T20:48:48.070 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 80: Connection refused 2023-10-05T20:48:50.071 DEBUG:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:48:50.097 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T20:48:50.097 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2023-10-05T20:48:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T20:48:50.098 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 80: Connection refused 2023-10-05T20:48:54.100 DEBUG:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:48:54.126 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T20:48:54.126 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2023-10-05T20:48:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T20:48:54.127 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi086.front.sepia.ceph.com port 80: Connection refused 2023-10-05T20:49:02.129 DEBUG:teuthology.orchestra.run.smithi086:> curl http://smithi086.front.sepia.ceph.com:80/ 2023-10-05T20:49:02.154 INFO:teuthology.orchestra.run.smithi086.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T20:49:02.155 INFO:teuthology.orchestra.run.smithi086.stderr: Dload Upload Total Spent Left Speed 2023-10-05T20:49:02.156 INFO:teuthology.orchestra.run.smithi086.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-05T20:49:02.157 INFO:teuthology.orchestra.run.smithi086.stdout:anonymous 2023-10-05T20:49:02.157 INFO:teuthology.run_tasks:Running task s3tests... 2023-10-05T20:49:02.171 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-10-05T20:49:02.172 INFO:tasks.s3tests:Downloading s3-tests... 2023-10-05T20:49:02.172 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-10-05T20:49:02.172 DEBUG:teuthology.orchestra.run.smithi086:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T20:49:02.217 INFO:teuthology.orchestra.run.smithi086.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2023-10-05T20:49:02.926 INFO:tasks.s3tests:Creating rgw users... 2023-10-05T20:49:02.927 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-10-05T20:49:02.927 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key KTOYILKNEKKGRDDHYUJN --secret /cpsJ47jwirBDtQv1WIfZbqaG/CagSRmYyRlOhp8HW2pltQQ4OjvEw== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T20:49:03.021 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:03.021 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:03.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.025+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 realm 2023-10-05T20:49:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.027+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.028+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.028+0000 7fad69d54680 20 searching for the correct realm 2023-10-05T20:49:03.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:03.037 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.035+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:03.038 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.039 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.036+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.042+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 zone default found 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 4 Realm: () 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 zonegroup default 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.043+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.044+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.044+0000 7fad69d54680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.044+0000 7fad69d54680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.044+0000 7fad69d54680 20 started sync module instance, tier type = 2023-10-05T20:49:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.044+0000 7fad69d54680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=0 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=1 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=2 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=3 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=4 2023-10-05T20:49:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=5 2023-10-05T20:49:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.060+0000 7fad69d54680 20 add_watcher() i=6 2023-10-05T20:49:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.065+0000 7fad69d54680 20 add_watcher() i=7 2023-10-05T20:49:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.065+0000 7fad69d54680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.067+0000 7fad2dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.067+0000 7fad69d54680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.067+0000 7fad69d54680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.070+0000 7fad69d54680 5 note: GC not initialized 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.071+0000 7faca4ff9700 20 reqs_thread_entry: start 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fac9a7fc700 20 reqs_thread_entry: start 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fac98ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:33 2023 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fac7ffff700 20 reqs_thread_entry: start 2023-10-05T20:49:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 20 get_system_obj_state: rctx=0x7fff5521bed8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5577e4d83240 s->prefetch_data=0 2023-10-05T20:49:03.075 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 20 get_system_obj_state: rctx=0x7fff5521bbc8 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x5577e4d83240 s->prefetch_data=0 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.072+0000 7fad69d54680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-10-05T20:49:03.076 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 20 get_system_obj_state: rctx=0x7fff5521bbc8 obj=default.rgw.meta:users.keys:KTOYILKNEKKGRDDHYUJN state=0x5577e4d83240 s->prefetch_data=0 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 cache get: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN : miss 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN info.flags=0x0 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 adding default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN to cache LRU end 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 20 get_system_obj_state: rctx=0x7fff5521b458 obj=default.rgw.meta:users.keys:KTOYILKNEKKGRDDHYUJN state=0x5577e4d83240 s->prefetch_data=0 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 cache get: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN : hit (negative entry) 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 20 get_system_obj_state: rctx=0x7fff5521bb08 obj=default.rgw.meta:users.keys:KTOYILKNEKKGRDDHYUJN state=0x5577e4d83240 s->prefetch_data=0 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.073+0000 7fad69d54680 10 cache get: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN : hit (negative entry) 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad69d54680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad69d54680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad567fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 93973430434864 notifier 4207 bl.length()=640 2023-10-05T20:49:03.077 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad567fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T20:49:03.078 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.074+0000 7fad567fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad69d54680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad69d54680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad55ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 93973430466256 notifier 4207 bl.length()=194 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad55ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:03.079 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.077+0000 7fad55ffb700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad69d54680 10 cache put: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN info.flags=0x7 2023-10-05T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad69d54680 10 moving default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN to cache LRU end 2023-10-05T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad69d54680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:KTOYILKNEKKGRDDHYUJN, ofs0, ns] 2023-10-05T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad567fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182401 cookie 93973430434864 notifier 4207 bl.length()=186 2023-10-05T20:49:03.080 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad567fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN info.flags=0x7 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.078+0000 7fad567fc700 10 rgw watcher librados: moving default.rgw.meta+users.keys+KTOYILKNEKKGRDDHYUJN to cache LRU end 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "user_id": "foo.client.0", 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "display_name": "Mr. foo.client.0", 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "email": "foo.client.0+test@test.test", 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "suspended": 0, 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "max_buckets": 1000, 2023-10-05T20:49:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "subusers": [], 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "keys": [ 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "foo.client.0", 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "KTOYILKNEKKGRDDHYUJN", 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "/cpsJ47jwirBDtQv1WIfZbqaG/CagSRmYyRlOhp8HW2pltQQ4OjvEw==" 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "swift_keys": [], 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": [ 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "user-policy", 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "perm": "*" 2023-10-05T20:49:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "op_mask": "read, write, delete", 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "default_storage_class": "", 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_tags": [], 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "temp_url_keys": [], 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "rgw", 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "mfa_ids": [] 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-10-05T20:49:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:49:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.081+0000 7fad69d54680 20 remove_watcher() i=0 2023-10-05T20:49:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.081+0000 7fad69d54680 2 removed watcher, disabling cache 2023-10-05T20:49:03.085 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.082+0000 7fad69d54680 20 remove_watcher() i=1 2023-10-05T20:49:03.086 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.084+0000 7fad69d54680 20 remove_watcher() i=2 2023-10-05T20:49:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.087+0000 7fad69d54680 20 remove_watcher() i=3 2023-10-05T20:49:03.091 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.090+0000 7fad69d54680 20 remove_watcher() i=4 2023-10-05T20:49:03.094 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.093+0000 7fad69d54680 20 remove_watcher() i=5 2023-10-05T20:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.095+0000 7fad69d54680 20 remove_watcher() i=6 2023-10-05T20:49:03.098 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.096+0000 7fad69d54680 20 remove_watcher() i=7 2023-10-05T20:49:03.098 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.097+0000 7fad2dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:03.103 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 4703549126 --totp-seed GA3DEXQWB6BPVXUU43OZ7BA4TB3OPWYMHNJLMDBZVZF3XA4L7IG2GS5T7ZEEKX52 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T20:49:03.156 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:03.156 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:03.162 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.160+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 20 realm 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.161+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.162+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.162+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.162+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.162+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.163+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.163+0000 7f93d36f7680 20 searching for the correct realm 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.170+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.171+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.171+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.171+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.177+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 zone default found 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 4 Realm: () 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:03.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 zonegroup default 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.178+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.179+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.179+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.179+0000 7f93d36f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.179+0000 7f93d36f7680 20 started sync module instance, tier type = 2023-10-05T20:49:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.179+0000 7f93d36f7680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=0 2023-10-05T20:49:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=1 2023-10-05T20:49:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=2 2023-10-05T20:49:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=3 2023-10-05T20:49:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=4 2023-10-05T20:49:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=5 2023-10-05T20:49:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.189+0000 7f93d36f7680 20 add_watcher() i=6 2023-10-05T20:49:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.190+0000 7f93d36f7680 20 add_watcher() i=7 2023-10-05T20:49:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.190+0000 7f93d36f7680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.192+0000 7f9397fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.192+0000 7f93d36f7680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.192+0000 7f93d36f7680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:03.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.196+0000 7f93d36f7680 5 note: GC not initialized 2023-10-05T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f9306ffd700 20 reqs_thread_entry: start 2023-10-05T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f9304ff9700 20 reqs_thread_entry: start 2023-10-05T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f93d36f7680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f93d36f7680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f92faffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:33 2023 2023-10-05T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f92fa7fc700 20 reqs_thread_entry: start 2023-10-05T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f93d36f7680 20 get_system_obj_state: rctx=0x7ffd62272f18 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55bdc09a9240 s->prefetch_data=0 2023-10-05T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.197+0000 7f93d36f7680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T20:49:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 20 Read xattr: user.rgw.idtag 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 20 rados->read ofs=0 len=0 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:03.198+0000 7f93d36f7680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93d36f7680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T20:49:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93d36f7680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93d36f7680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T20:49:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93b77fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94273470143200 notifier 4213 bl.length()=654 2023-10-05T20:49:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93b77fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T20:49:05.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.161+0000 7f93b77fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:49:05.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.163+0000 7f93d36f7680 20 remove_watcher() i=0 2023-10-05T20:49:05.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.163+0000 7f93d36f7680 2 removed watcher, disabling cache 2023-10-05T20:49:05.165 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.164+0000 7f93d36f7680 20 remove_watcher() i=1 2023-10-05T20:49:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.167+0000 7f93d36f7680 20 remove_watcher() i=2 2023-10-05T20:49:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.172+0000 7f93d36f7680 20 remove_watcher() i=3 2023-10-05T20:49:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.173+0000 7f93d36f7680 20 remove_watcher() i=4 2023-10-05T20:49:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.176+0000 7f93d36f7680 20 remove_watcher() i=5 2023-10-05T20:49:05.179 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.177+0000 7f93d36f7680 20 remove_watcher() i=6 2023-10-05T20:49:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.178+0000 7f93d36f7680 20 remove_watcher() i=7 2023-10-05T20:49:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.179+0000 7f9397fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.186 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-10-05T20:49:05.186 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key TGPNIQLUVGCFOOTSCTXL --secret AObVQUuTHbaYy0pf5pg59ErLtKiTlKtbwr/L5kExpo6/KDSi51PKgg== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T20:49:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.245+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 realm 2023-10-05T20:49:05.248 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.246+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.247+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.247+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.247+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.250 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.247+0000 7fe5e745a680 20 searching for the correct realm 2023-10-05T20:49:05.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.258 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.261 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.255+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.256+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.256+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.262 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.256+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.262+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.267 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 zone default found 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 4 Realm: () 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 zonegroup default 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.263+0000 7fe5e745a680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.268 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.264+0000 7fe5e745a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.264+0000 7fe5e745a680 20 started sync module instance, tier type = 2023-10-05T20:49:05.269 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.264+0000 7fe5e745a680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:05.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=0 2023-10-05T20:49:05.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=1 2023-10-05T20:49:05.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=2 2023-10-05T20:49:05.299 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=3 2023-10-05T20:49:05.300 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=4 2023-10-05T20:49:05.300 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=5 2023-10-05T20:49:05.300 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.296+0000 7fe5e745a680 20 add_watcher() i=6 2023-10-05T20:49:05.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.299+0000 7fe5e745a680 20 add_watcher() i=7 2023-10-05T20:49:05.301 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.299+0000 7fe5e745a680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.302+0000 7fe5abfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.302+0000 7fe5e745a680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.302+0000 7fe5e745a680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.306+0000 7fe5e745a680 5 note: GC not initialized 2023-10-05T20:49:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe51affd700 20 reqs_thread_entry: start 2023-10-05T20:49:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe518ff9700 20 reqs_thread_entry: start 2023-10-05T20:49:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe5e745a680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe5e745a680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe50effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:35 2023 2023-10-05T20:49:05.312 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe50e7fc700 20 reqs_thread_entry: start 2023-10-05T20:49:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe5e745a680 20 get_system_obj_state: rctx=0x7ffeaf4372f8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55b9c7819240 s->prefetch_data=0 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.307+0000 7fe5e745a680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 20 get_system_obj_state: rctx=0x7ffeaf436fe8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55b9c7819240 s->prefetch_data=0 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-10-05T20:49:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-10-05T20:49:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 20 get_system_obj_state: rctx=0x7ffeaf436fe8 obj=default.rgw.meta:users.keys:TGPNIQLUVGCFOOTSCTXL state=0x55b9c7819240 s->prefetch_data=0 2023-10-05T20:49:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache get: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL : miss 2023-10-05T20:49:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL info.flags=0x0 2023-10-05T20:49:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 adding default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL to cache LRU end 2023-10-05T20:49:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 20 get_system_obj_state: rctx=0x7ffeaf436878 obj=default.rgw.meta:users.keys:TGPNIQLUVGCFOOTSCTXL state=0x55b9c7819240 s->prefetch_data=0 2023-10-05T20:49:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache get: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL : hit (negative entry) 2023-10-05T20:49:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 20 get_system_obj_state: rctx=0x7ffeaf436f28 obj=default.rgw.meta:users.keys:TGPNIQLUVGCFOOTSCTXL state=0x55b9c7819240 s->prefetch_data=0 2023-10-05T20:49:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.308+0000 7fe5e745a680 10 cache get: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL : hit (negative entry) 2023-10-05T20:49:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T20:49:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5e745a680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5e745a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T20:49:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94256406078608 notifier 4219 bl.length()=640 2023-10-05T20:49:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5cbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T20:49:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.311+0000 7fe5cbfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5e745a680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5e745a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5cb7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94256406078608 notifier 4219 bl.length()=194 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5cb7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.313+0000 7fe5cb7fe700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5e745a680 10 cache put: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL info.flags=0x7 2023-10-05T20:49:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5e745a680 10 moving default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL to cache LRU end 2023-10-05T20:49:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5e745a680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:TGPNIQLUVGCFOOTSCTXL, ofs0, ns] 2023-10-05T20:49:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94256406086016 notifier 4219 bl.length()=186 2023-10-05T20:49:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5cbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL info.flags=0x7 2023-10-05T20:49:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.315+0000 7fe5cbfff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+TGPNIQLUVGCFOOTSCTXL to cache LRU end 2023-10-05T20:49:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-10-05T20:49:05.321 INFO:teuthology.orchestra.run.smithi086.stdout: "user_id": "bar.client.0", 2023-10-05T20:49:05.321 INFO:teuthology.orchestra.run.smithi086.stdout: "display_name": "Mr. bar.client.0", 2023-10-05T20:49:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "email": "bar.client.0+test@test.test", 2023-10-05T20:49:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "suspended": 0, 2023-10-05T20:49:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "max_buckets": 1000, 2023-10-05T20:49:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "subusers": [], 2023-10-05T20:49:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "keys": [ 2023-10-05T20:49:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "bar.client.0", 2023-10-05T20:49:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "TGPNIQLUVGCFOOTSCTXL", 2023-10-05T20:49:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "AObVQUuTHbaYy0pf5pg59ErLtKiTlKtbwr/L5kExpo6/KDSi51PKgg==" 2023-10-05T20:49:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "swift_keys": [], 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": [ 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "user-policy", 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "perm": "*" 2023-10-05T20:49:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "op_mask": "read, write, delete", 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "default_storage_class": "", 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_tags": [], 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "temp_url_keys": [], 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "rgw", 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "mfa_ids": [] 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-10-05T20:49:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:49:05.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.316+0000 7fe5e745a680 20 remove_watcher() i=0 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.316+0000 7fe5e745a680 2 removed watcher, disabling cache 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.317+0000 7fe5e745a680 20 remove_watcher() i=1 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.318+0000 7fe5e745a680 20 remove_watcher() i=2 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.319+0000 7fe5e745a680 20 remove_watcher() i=3 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.320+0000 7fe5e745a680 20 remove_watcher() i=4 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.322+0000 7fe5e745a680 20 remove_watcher() i=5 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.323+0000 7fe5e745a680 20 remove_watcher() i=6 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.326+0000 7fe5e745a680 20 remove_watcher() i=7 2023-10-05T20:49:05.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.327+0000 7fe5abfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.333 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 9359139462 --totp-seed SZP2GXPN4CED6OEDD3SMXDJC6S4BUK34NGITYSIS5KA43ADEBEF5H6VRYGG5KYUK --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T20:49:05.384 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:05.384 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.388+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 20 realm 2023-10-05T20:49:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.389+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.390+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.390+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.390+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.390+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.390+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.393 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.391+0000 7fe77a20f680 20 searching for the correct realm 2023-10-05T20:49:05.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.397+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.397+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.397+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.400 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.397+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.397+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.398+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.399+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.405+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 zone default found 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 4 Realm: () 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 zonegroup default 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 started sync module instance, tier type = 2023-10-05T20:49:05.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.406+0000 7fe77a20f680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=0 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=1 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=2 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=3 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=4 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=5 2023-10-05T20:49:05.429 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.427+0000 7fe77a20f680 20 add_watcher() i=6 2023-10-05T20:49:05.432 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.431+0000 7fe77a20f680 20 add_watcher() i=7 2023-10-05T20:49:05.433 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.431+0000 7fe77a20f680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:05.434 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.432+0000 7fe73e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.434 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.432+0000 7fe77a20f680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:05.434 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.432+0000 7fe77a20f680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:05.438 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.436+0000 7fe77a20f680 5 note: GC not initialized 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.437+0000 7fe6ad7fa700 20 reqs_thread_entry: start 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.437+0000 7fe6a6ffd700 20 reqs_thread_entry: start 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.437+0000 7fe77a20f680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe6a57fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:35 2023 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe6a4ff9700 20 reqs_thread_entry: start 2023-10-05T20:49:05.440 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 20 get_system_obj_state: rctx=0x7ffcc2002a78 obj=default.rgw.meta:users.uid:bar.client.0 state=0x560c6d029240 s->prefetch_data=0 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 20 Read xattr: user.rgw.idtag 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:49:05.441 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T20:49:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T20:49:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.438+0000 7fe77a20f680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:05.443 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.441+0000 7fe77a20f680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T20:49:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.441+0000 7fe77a20f680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.441+0000 7fe77a20f680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T20:49:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.442+0000 7fe75effd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94611370092720 notifier 4225 bl.length()=654 2023-10-05T20:49:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.442+0000 7fe75effd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T20:49:05.444 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.442+0000 7fe75effd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:49:05.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.445+0000 7fe77a20f680 20 remove_watcher() i=0 2023-10-05T20:49:05.447 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.445+0000 7fe77a20f680 2 removed watcher, disabling cache 2023-10-05T20:49:05.448 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.446+0000 7fe77a20f680 20 remove_watcher() i=1 2023-10-05T20:49:05.449 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.448+0000 7fe77a20f680 20 remove_watcher() i=2 2023-10-05T20:49:05.452 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.450+0000 7fe77a20f680 20 remove_watcher() i=3 2023-10-05T20:49:05.453 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.452+0000 7fe77a20f680 20 remove_watcher() i=4 2023-10-05T20:49:05.455 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.454+0000 7fe77a20f680 20 remove_watcher() i=5 2023-10-05T20:49:05.457 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.456+0000 7fe77a20f680 20 remove_watcher() i=6 2023-10-05T20:49:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.456+0000 7fe77a20f680 20 remove_watcher() i=7 2023-10-05T20:49:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.457+0000 7fe73e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.463 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-10-05T20:49:05.464 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key HPMVCDIDZBCEPDLRLFZJ --secret pCe8gxRW8uOstLMc665FY+d4IwcIfWkmJYxdsjIPhUr2finrEBbiXA== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-10-05T20:49:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:05.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.520+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 realm 2023-10-05T20:49:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.521+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.522+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.522+0000 7f4269b50680 20 searching for the correct realm 2023-10-05T20:49:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.528+0000 7f4269b50680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.528+0000 7f4269b50680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.528+0000 7f4269b50680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.528+0000 7f4269b50680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.528+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.529+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.533 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.530+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.535+0000 7f4269b50680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.535+0000 7f4269b50680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.535+0000 7f4269b50680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.535+0000 7f4269b50680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.535+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.536+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 zone default found 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 4 Realm: () 2023-10-05T20:49:05.540 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 zonegroup default 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 started sync module instance, tier type = 2023-10-05T20:49:05.541 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.537+0000 7f4269b50680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:05.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=0 2023-10-05T20:49:05.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=1 2023-10-05T20:49:05.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=2 2023-10-05T20:49:05.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=3 2023-10-05T20:49:05.563 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=4 2023-10-05T20:49:05.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=5 2023-10-05T20:49:05.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.561+0000 7f4269b50680 20 add_watcher() i=6 2023-10-05T20:49:05.564 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.563+0000 7f4269b50680 20 add_watcher() i=7 2023-10-05T20:49:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.563+0000 7f4269b50680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:05.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.565+0000 7f422dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.565+0000 7f4269b50680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:05.567 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.565+0000 7f4269b50680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.569+0000 7f4269b50680 5 note: GC not initialized 2023-10-05T20:49:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f41a4ff9700 20 reqs_thread_entry: start 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f419a7fc700 20 reqs_thread_entry: start 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f4269b50680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f4269b50680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f4198ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:35 2023 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f417ffff700 20 reqs_thread_entry: start 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f4269b50680 20 get_system_obj_state: rctx=0x7ffcbc3f2dd8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55dc4dc75240 s->prefetch_data=0 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.570+0000 7f4269b50680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 20 get_system_obj_state: rctx=0x7ffcbc3f2ac8 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x55dc4dc75240 s->prefetch_data=0 2023-10-05T20:49:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-10-05T20:49:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-10-05T20:49:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 20 get_system_obj_state: rctx=0x7ffcbc3f2ac8 obj=default.rgw.meta:users.keys:HPMVCDIDZBCEPDLRLFZJ state=0x55dc4dc75240 s->prefetch_data=0 2023-10-05T20:49:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache get: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ : miss 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ info.flags=0x0 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 adding default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ to cache LRU end 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 20 get_system_obj_state: rctx=0x7ffcbc3f2358 obj=default.rgw.meta:users.keys:HPMVCDIDZBCEPDLRLFZJ state=0x55dc4dc75240 s->prefetch_data=0 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache get: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ : hit (negative entry) 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 20 get_system_obj_state: rctx=0x7ffcbc3f2a08 obj=default.rgw.meta:users.keys:HPMVCDIDZBCEPDLRLFZJ state=0x55dc4dc75240 s->prefetch_data=0 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.571+0000 7f4269b50680 10 cache get: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ : hit (negative entry) 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f4269b50680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f4269b50680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T20:49:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f42567fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94404687704224 notifier 4231 bl.length()=714 2023-10-05T20:49:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f42567fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T20:49:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.572+0000 7f42567fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:05.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4269b50680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4269b50680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T20:49:05.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4255ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94404687710288 notifier 4231 bl.length()=224 2023-10-05T20:49:05.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4255ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T20:49:05.577 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.574+0000 7f4255ffb700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f4269b50680 10 cache put: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ info.flags=0x7 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f4269b50680 10 moving default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ to cache LRU end 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f4269b50680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:HPMVCDIDZBCEPDLRLFZJ, ofs0, ns] 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f42567fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94404687692288 notifier 4231 bl.length()=201 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f42567fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ info.flags=0x7 2023-10-05T20:49:05.578 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.576+0000 7f42567fc700 10 rgw watcher librados: moving default.rgw.meta+users.keys+HPMVCDIDZBCEPDLRLFZJ to cache LRU end 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "user_id": "testx$tenanteduser.client.0", 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "suspended": 0, 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "max_buckets": 1000, 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "subusers": [], 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "keys": [ 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "user": "testx$tenanteduser.client.0", 2023-10-05T20:49:05.579 INFO:teuthology.orchestra.run.smithi086.stdout: "access_key": "HPMVCDIDZBCEPDLRLFZJ", 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: "secret_key": "pCe8gxRW8uOstLMc665FY+d4IwcIfWkmJYxdsjIPhUr2finrEBbiXA==" 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: "swift_keys": [], 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": [ 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "user-policy", 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: "perm": "*" 2023-10-05T20:49:05.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: "op_mask": "read, write, delete", 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: "default_placement": "", 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: "default_storage_class": "", 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: "placement_tags": [], 2023-10-05T20:49:05.581 INFO:teuthology.orchestra.run.smithi086.stdout: "bucket_quota": { 2023-10-05T20:49:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:05.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:05.583 INFO:teuthology.orchestra.run.smithi086.stdout: "user_quota": { 2023-10-05T20:49:05.583 INFO:teuthology.orchestra.run.smithi086.stdout: "enabled": false, 2023-10-05T20:49:05.583 INFO:teuthology.orchestra.run.smithi086.stdout: "check_on_raw": false, 2023-10-05T20:49:05.583 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size": -1, 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: "max_size_kb": 0, 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: "max_objects": -1 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: "temp_url_keys": [], 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "rgw", 2023-10-05T20:49:05.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mfa_ids": [] 2023-10-05T20:49:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:} 2023-10-05T20:49:05.585 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:49:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.577+0000 7f4269b50680 20 remove_watcher() i=0 2023-10-05T20:49:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.577+0000 7f4269b50680 2 removed watcher, disabling cache 2023-10-05T20:49:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.578+0000 7f4269b50680 20 remove_watcher() i=1 2023-10-05T20:49:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.579+0000 7f4269b50680 20 remove_watcher() i=2 2023-10-05T20:49:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.580+0000 7f4269b50680 20 remove_watcher() i=3 2023-10-05T20:49:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.581+0000 7f4269b50680 20 remove_watcher() i=4 2023-10-05T20:49:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.582+0000 7f4269b50680 20 remove_watcher() i=5 2023-10-05T20:49:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.582+0000 7f4269b50680 20 remove_watcher() i=6 2023-10-05T20:49:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.583+0000 7f4269b50680 20 remove_watcher() i=7 2023-10-05T20:49:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.584+0000 7f422dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.590 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 9823803778 --totp-seed PVPYCGZKRNQDEBXCG7DAYETDXF6ZIEU4UYFGPURBB22DU46KWI5H34ZNJO5MJ3RA --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T20:49:05.641 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:49:05.641 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:49:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.645+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 20 realm 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:49:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.646+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.647+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.647+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.647+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.647+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.648+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.648+0000 7f423c7a9680 20 searching for the correct realm 2023-10-05T20:49:05.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.657 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.654+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.658 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.659 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.655+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.660 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.656+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.661+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.661+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.661+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.661+0000 7f423c7a9680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.661+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.666 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 zone default found 2023-10-05T20:49:05.667 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 4 Realm: () 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 zonegroup default 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.662+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.668 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:49:05.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 started sync module instance, tier type = 2023-10-05T20:49:05.669 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.663+0000 7f423c7a9680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=0 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=1 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=2 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=3 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=4 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=5 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.670+0000 7f423c7a9680 20 add_watcher() i=6 2023-10-05T20:49:05.674 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.671+0000 7f423c7a9680 20 add_watcher() i=7 2023-10-05T20:49:05.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.671+0000 7f423c7a9680 2 all 8 watchers are set, enabling cache 2023-10-05T20:49:05.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.672+0000 7f42017e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.672+0000 7f423c7a9680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:49:05.675 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.672+0000 7f423c7a9680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:49:05.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.676+0000 7f423c7a9680 5 note: GC not initialized 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f416ffff700 20 reqs_thread_entry: start 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f416dffb700 20 reqs_thread_entry: start 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f423c7a9680 20 init_complete bucket index max shards: 11 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f423c7a9680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f415bfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:49:35 2023 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f415b7fe700 20 reqs_thread_entry: start 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f423c7a9680 20 get_system_obj_state: rctx=0x7ffe589ff6d8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x558d6e742240 s->prefetch_data=0 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.677+0000 7f423c7a9680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T20:49:05.680 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 20 Read xattr: user.rgw.idtag 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 20 rados->read ofs=0 len=0 2023-10-05T20:49:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 20 rados_obj.operate() r=0 bl.length=514 2023-10-05T20:49:05.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T20:49:05.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.678+0000 7f423c7a9680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:49:05.682 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.680+0000 7f423c7a9680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T20:49:05.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.680+0000 7f423c7a9680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.680+0000 7f423c7a9680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T20:49:05.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.681+0000 7f422d87d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94065933482032 notifier 4237 bl.length()=728 2023-10-05T20:49:05.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.681+0000 7f422d87d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T20:49:05.683 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.681+0000 7f422d87d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:49:05.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.682+0000 7f423c7a9680 20 remove_watcher() i=0 2023-10-05T20:49:05.684 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.682+0000 7f423c7a9680 2 removed watcher, disabling cache 2023-10-05T20:49:05.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.683+0000 7f423c7a9680 20 remove_watcher() i=1 2023-10-05T20:49:05.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.684+0000 7f423c7a9680 20 remove_watcher() i=2 2023-10-05T20:49:05.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.685+0000 7f423c7a9680 20 remove_watcher() i=3 2023-10-05T20:49:05.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.685+0000 7f423c7a9680 20 remove_watcher() i=4 2023-10-05T20:49:05.688 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.686+0000 7f423c7a9680 20 remove_watcher() i=5 2023-10-05T20:49:05.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.687+0000 7f423c7a9680 20 remove_watcher() i=6 2023-10-05T20:49:05.689 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.688+0000 7f423c7a9680 20 remove_watcher() i=7 2023-10-05T20:49:05.691 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:49:05.688+0000 7f42017e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:49:05.695 INFO:tasks.s3tests:Configuring s3-tests... 2023-10-05T20:49:05.695 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2023-10-05T20:49:14.748 INFO:teuthology.orchestra.run.smithi086.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-10-05T20:49:15.595 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:44:37 ago on Thu 05 Oct 2023 08:04:38 PM UTC. 2023-10-05T20:49:17.170 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-10-05T20:49:17.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:49:17.171 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-10-05T20:49:17.171 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: libffi-devel x86_64 3.1-24.el8 baseos 29 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: libxml2-devel x86_64 2.9.7-16.el8 appstream 1.0 M 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: cmake-filesystem x86_64 3.20.2-5.el8 appstream 45 k 2023-10-05T20:49:17.172 INFO:teuthology.orchestra.run.smithi086.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout:Install 9 Packages 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 1 Package 2023-10-05T20:49:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:49:17.174 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2023-10-05T20:49:17.174 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-10-05T20:54:18.551 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] cmake-filesystem-3.20.2-5.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/cmake-filesystem-3.20.2-5.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:54:18.552 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libevent-devel-2.1.8-5.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/libevent-devel-2.1.8-5.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:54:18.552 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libxml2-devel-2.9.7-16.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/libxml2-devel-2.9.7-16.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-10-05T20:54:18.991 INFO:teuthology.orchestra.run.smithi086.stdout:(1/10): cmake-filesystem-3.20.2-5.el8.x86_64.rp 151 B/s | 45 kB 05:01 2023-10-05T20:54:19.024 INFO:teuthology.orchestra.run.smithi086.stdout:(2/10): libevent-devel-2.1.8-5.el8.x86_64.rpm 354 B/s | 104 kB 05:01 2023-10-05T20:54:19.075 INFO:teuthology.orchestra.run.smithi086.stdout:(3/10): libffi-devel-3.1-24.el8.x86_64.rpm 583 kB/s | 29 kB 00:00 2023-10-05T20:54:19.125 INFO:teuthology.orchestra.run.smithi086.stdout:(4/10): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 2.9 MB/s | 150 kB 00:00 2023-10-05T20:54:19.159 INFO:teuthology.orchestra.run.smithi086.stdout:(5/10): libgpg-error-devel-1.31-1.el8.x86_64.rp 1.9 MB/s | 64 kB 00:00 2023-10-05T20:54:19.192 INFO:teuthology.orchestra.run.smithi086.stdout:(6/10): xz-devel-5.2.4-4.el8.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-05T20:54:19.226 INFO:teuthology.orchestra.run.smithi086.stdout:(7/10): zlib-devel-1.2.11-25.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2023-10-05T20:54:19.259 INFO:teuthology.orchestra.run.smithi086.stdout:(8/10): zlib-1.2.11-25.el8.x86_64.rpm 3.0 MB/s | 103 kB 00:00 2023-10-05T20:54:19.318 INFO:teuthology.orchestra.run.smithi086.stdout:(9/10): libxml2-devel-2.9.7-16.el8.x86_64.rpm 3.5 kB/s | 1.0 MB 05:01 2023-10-05T20:54:19.360 INFO:teuthology.orchestra.run.smithi086.stdout:(10/10): libxslt-devel-1.1.32-6.el8.x86_64.rpm 876 kB/s | 323 kB 00:00 2023-10-05T20:54:19.360 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-10-05T20:54:19.360 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.6 kB/s | 2.0 MB 05:02 2023-10-05T20:54:19.452 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-10-05T20:54:19.581 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-10-05T20:54:19.581 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-10-05T20:54:19.701 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-10-05T20:54:19.704 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-10-05T20:54:20.089 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-10-05T20:54:20.222 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/11 2023-10-05T20:54:20.256 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/11 2023-10-05T20:54:20.479 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 2/11 2023-10-05T20:54:20.495 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 2/11 2023-10-05T20:54:20.678 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 3/11 2023-10-05T20:54:20.860 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 4/11 2023-10-05T20:54:21.011 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 5/11 2023-10-05T20:54:21.431 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cmake-filesystem-3.20.2-5.el8.x86_64 6/11 2023-10-05T20:54:21.744 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libxml2-devel-2.9.7-16.el8.x86_64 7/11 2023-10-05T20:54:21.960 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 8/11 2023-10-05T20:54:22.089 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 9/11 2023-10-05T20:54:22.116 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 9/11 2023-10-05T20:54:22.315 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 10/11 2023-10-05T20:54:22.471 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cmake-filesystem-3.20.2-5.el8.x86_64 1/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libxml2-devel-2.9.7-16.el8.x86_64 3/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/11 2023-10-05T20:54:22.775 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/11 2023-10-05T20:54:22.776 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 9/11 2023-10-05T20:54:22.776 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 10/11 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 11/11 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-1.2.11-25.el8.x86_64 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: cmake-filesystem-3.20.2-5.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-16.el8.x86_64 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2023-10-05T20:54:23.482 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel-1.2.11-25.el8.x86_64 2023-10-05T20:54:23.483 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:23.483 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-10-05T20:54:24.066 INFO:teuthology.orchestra.run.smithi086.stdout:Already using interpreter /usr/bin/python3.6 2023-10-05T20:54:24.066 INFO:teuthology.orchestra.run.smithi086.stdout:Using base prefix '/usr' 2023-10-05T20:54:24.066 INFO:teuthology.orchestra.run.smithi086.stdout:New python executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python3.6 2023-10-05T20:54:24.066 INFO:teuthology.orchestra.run.smithi086.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python 2023-10-05T20:54:27.738 INFO:teuthology.orchestra.run.smithi086.stdout:Installing setuptools, pip, wheel...done. 2023-10-05T20:54:28.079 INFO:teuthology.orchestra.run.smithi086.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-10-05T20:54:29.300 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting setuptools==44.1.0 2023-10-05T20:54:29.364 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-10-05T20:54:29.488 INFO:teuthology.orchestra.run.smithi086.stdout:Installing collected packages: setuptools 2023-10-05T20:54:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: Attempting uninstall: setuptools 2023-10-05T20:54:29.488 INFO:teuthology.orchestra.run.smithi086.stdout: Found existing installation: setuptools 59.6.0 2023-10-05T20:54:29.534 INFO:teuthology.orchestra.run.smithi086.stdout: Uninstalling setuptools-59.6.0: 2023-10-05T20:54:29.543 INFO:teuthology.orchestra.run.smithi086.stdout: Successfully uninstalled setuptools-59.6.0 2023-10-05T20:54:29.765 INFO:teuthology.orchestra.run.smithi086.stdout:Successfully installed setuptools-44.1.0 2023-10-05T20:54:30.360 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting PyYAML 2023-10-05T20:54:30.418 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2023-10-05T20:54:30.562 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting nose>=1.0.0 2023-10-05T20:54:30.581 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-10-05T20:54:30.691 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting boto>=2.6.0 2023-10-05T20:54:30.711 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-10-05T20:54:31.694 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting boto3>=1.0.0 2023-10-05T20:54:31.712 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-10-05T20:54:32.807 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting botocore<1.28.0 2023-10-05T20:54:32.827 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-10-05T20:54:33.082 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting munch>=2.0.0 2023-10-05T20:54:33.100 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-10-05T20:54:33.539 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting gevent>=1.0 2023-10-05T20:54:33.560 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-10-05T20:54:33.733 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting isodate>=0.4.4 2023-10-05T20:54:33.750 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-10-05T20:54:33.856 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting requests>=2.23.0 2023-10-05T20:54:33.874 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-10-05T20:54:34.059 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting pytz>=2011k 2023-10-05T20:54:34.077 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T20:54:34.182 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting httplib2 2023-10-05T20:54:34.200 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-10-05T20:54:34.743 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting lxml 2023-10-05T20:54:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl (7.4 MB) 2023-10-05T20:54:35.001 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-10-05T20:54:35.018 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-10-05T20:54:35.108 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-10-05T20:54:35.127 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-10-05T20:54:35.317 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-10-05T20:54:35.334 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-10-05T20:54:35.467 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting urllib3<1.27,>=1.25.4 2023-10-05T20:54:35.484 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading urllib3-1.26.17-py2.py3-none-any.whl (143 kB) 2023-10-05T20:54:35.683 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting importlib-metadata>=1.7.0 2023-10-05T20:54:35.700 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2023-10-05T20:54:35.817 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting zope.event 2023-10-05T20:54:35.835 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-10-05T20:54:35.856 INFO:teuthology.orchestra.run.smithi086.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-10-05T20:54:36.295 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting greenlet>=2.0.0 2023-10-05T20:54:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2023-10-05T20:54:36.724 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting zope.interface 2023-10-05T20:54:36.744 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-10-05T20:54:36.825 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting six 2023-10-05T20:54:36.842 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T20:54:37.112 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting charset-normalizer~=2.0.0 2023-10-05T20:54:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-10-05T20:54:37.269 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting certifi>=2017.4.17 2023-10-05T20:54:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T20:54:37.378 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting idna<4,>=2.5 2023-10-05T20:54:37.395 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T20:54:37.552 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-10-05T20:54:37.569 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-10-05T20:54:37.711 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting typing-extensions>=3.6.4 2023-10-05T20:54:37.728 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2023-10-05T20:54:37.832 INFO:teuthology.orchestra.run.smithi086.stdout:Collecting zipp>=0.5 2023-10-05T20:54:37.849 INFO:teuthology.orchestra.run.smithi086.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2023-10-05T20:54:38.212 INFO:teuthology.orchestra.run.smithi086.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-10-05T20:54:41.743 INFO:teuthology.orchestra.run.smithi086.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.7.22 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.4 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.17 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2023-10-05T20:54:42.051 INFO:teuthology.orchestra.run.smithi086.stdout:running develop 2023-10-05T20:54:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:running egg_info 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:creating s3tests.egg-info 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:writing s3tests.egg-info/PKG-INFO 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-10-05T20:54:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T20:54:42.082 INFO:teuthology.orchestra.run.smithi086.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T20:54:42.083 INFO:teuthology.orchestra.run.smithi086.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T20:54:42.090 INFO:teuthology.orchestra.run.smithi086.stdout:running build_ext 2023-10-05T20:54:42.090 INFO:teuthology.orchestra.run.smithi086.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-10-05T20:54:42.090 INFO:teuthology.orchestra.run.smithi086.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-10-05T20:54:42.091 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.091 INFO:teuthology.orchestra.run.smithi086.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T20:54:42.091 INFO:teuthology.orchestra.run.smithi086.stdout:Processing dependencies for s3tests==0.0.1 2023-10-05T20:54:42.199 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for isodate==0.6.1 2023-10-05T20:54:42.199 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: isodate 0.6.1 2023-10-05T20:54:42.199 INFO:teuthology.orchestra.run.smithi086.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-10-05T20:54:42.199 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.199 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.200 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for gevent==22.10.2 2023-10-05T20:54:42.200 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: gevent 22.10.2 2023-10-05T20:54:42.200 INFO:teuthology.orchestra.run.smithi086.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-10-05T20:54:42.200 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for munch==4.0.0 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: munch 4.0.0 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.201 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for PyYAML==6.0.1 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: PyYAML 6.0.1 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for boto3==1.23.10 2023-10-05T20:54:42.202 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: boto3 1.23.10 2023-10-05T20:54:42.203 INFO:teuthology.orchestra.run.smithi086.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-10-05T20:54:42.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.203 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for boto==2.49.0 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: boto 2.49.0 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for six==1.16.0 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: six 1.16.0 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Adding six 1.16.0 to easy-install.pth file 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for setuptools==44.1.0 2023-10-05T20:54:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: setuptools 44.1.0 2023-10-05T20:54:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-10-05T20:54:42.205 INFO:teuthology.orchestra.run.smithi086.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T20:54:42.206 INFO:teuthology.orchestra.run.smithi086.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T20:54:42.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.206 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.207 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for greenlet==2.0.2 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: greenlet 2.0.2 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for zope.interface==5.5.2 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: zope.interface 5.5.2 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.210 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for zope.event==4.6 2023-10-05T20:54:42.210 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: zope.event 4.6 2023-10-05T20:54:42.210 INFO:teuthology.orchestra.run.smithi086.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for importlib-metadata==4.8.3 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: importlib-metadata 4.8.3 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for botocore==1.26.10 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: botocore 1.26.10 2023-10-05T20:54:42.211 INFO:teuthology.orchestra.run.smithi086.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-10-05T20:54:42.212 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.212 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.214 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for s3transfer==0.5.2 2023-10-05T20:54:42.214 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: s3transfer 0.5.2 2023-10-05T20:54:42.214 INFO:teuthology.orchestra.run.smithi086.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for jmespath==0.10.0 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: jmespath 0.10.0 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for typing-extensions==4.1.1 2023-10-05T20:54:42.215 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: typing-extensions 4.1.1 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for zipp==3.6.0 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: zipp 3.6.0 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Adding zipp 3.6.0 to easy-install.pth file 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for urllib3==1.26.17 2023-10-05T20:54:42.216 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: urllib3 1.26.17 2023-10-05T20:54:42.217 INFO:teuthology.orchestra.run.smithi086.stdout:Adding urllib3 1.26.17 to easy-install.pth file 2023-10-05T20:54:42.217 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.217 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.217 INFO:teuthology.orchestra.run.smithi086.stdout:Searching for python-dateutil==2.8.2 2023-10-05T20:54:42.218 INFO:teuthology.orchestra.run.smithi086.stdout:Best match: python-dateutil 2.8.2 2023-10-05T20:54:42.218 INFO:teuthology.orchestra.run.smithi086.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-10-05T20:54:42.218 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:54:42.218 INFO:teuthology.orchestra.run.smithi086.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T20:54:42.218 INFO:teuthology.orchestra.run.smithi086.stdout:Finished processing dependencies for s3tests==0.0.1 2023-10-05T20:54:42.235 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:54:42.235 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-10-05T20:54:42.253 INFO:tasks.s3tests:Configuring boto... 2023-10-05T20:54:42.253 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-10-05T20:54:42.253 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T20:54:42.311 DEBUG:teuthology.orchestra.run.smithi086:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-10-05T20:54:42.790 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-10-05T20:54:42.845 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-10-05T20:54:42.901 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-10-05T20:54:42.954 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-10-05T20:54:43.007 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-10-05T20:54:43.026 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-10-05T20:54:43.086 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-10-05T20:54:43.097 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-10-05T20:54:43.139 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-10-05T20:54:43.181 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-10-05T20:54:43.231 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-10-05T20:54:43.290 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-10-05T20:54:43.346 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T20:54:43.403 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-10-05T20:54:43.404 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T20:54:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T20:54:43.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-10-05T20:54:43.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T20:54:43.452 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T20:54:43.596 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-10-05T20:54:43.596 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-10-05T20:54:43.596 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-10-05T20:54:43.596 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-10-05T20:54:43.596 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-10-05T20:54:43.597 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-10-05T20:54:43.597 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-10-05T20:54:43.698 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-10-05T20:54:43.746 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-10-05T20:54:45.320 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-10-05T20:54:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-10-05T20:54:45.492 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-10-05T20:54:45.527 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T20:54:45.620 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-10-05T20:54:45.621 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-10-05T20:54:45.622 INFO:teuthology.orchestra.run.smithi086.stderr:SKIP 2023-10-05T20:54:47.309 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-10-05T20:54:47.309 INFO:teuthology.orchestra.run.smithi086.stderr:/home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-10-05T20:54:47.309 INFO:teuthology.orchestra.run.smithi086.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-10-05T20:54:47.337 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-10-05T20:54:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-10-05T20:54:59.176 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-10-05T20:54:59.217 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-10-05T20:54:59.258 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-10-05T20:54:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-10-05T20:54:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-10-05T20:54:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-10-05T20:54:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-10-05T20:54:59.458 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-10-05T20:54:59.486 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-10-05T20:54:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-10-05T20:54:59.531 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-10-05T20:54:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-10-05T20:54:59.554 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-10-05T20:54:59.580 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-10-05T20:54:59.612 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-10-05T20:54:59.643 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-10-05T20:54:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-10-05T20:54:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-10-05T20:54:59.719 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-10-05T20:54:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T20:54:59.773 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-10-05T20:54:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-10-05T20:54:59.806 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-10-05T20:54:59.814 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-10-05T20:54:59.823 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-10-05T20:54:59.831 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-10-05T20:54:59.839 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-10-05T20:54:59.847 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T20:54:59.928 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-10-05T20:55:00.017 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-10-05T20:55:00.109 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-10-05T20:55:00.202 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-10-05T20:55:00.292 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-10-05T20:55:00.371 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-10-05T20:55:00.425 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-10-05T20:55:00.482 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-10-05T20:55:00.538 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-10-05T20:55:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-10-05T20:55:00.779 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-10-05T20:55:00.815 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T20:55:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T20:55:00.911 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-10-05T20:55:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-10-05T20:55:01.110 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-10-05T20:55:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-10-05T20:55:01.286 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-10-05T20:55:01.337 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-10-05T20:55:01.394 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-10-05T20:55:01.447 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-10-05T20:55:01.503 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-10-05T20:55:01.561 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-10-05T20:55:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-10-05T20:55:01.675 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-10-05T20:55:01.731 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-10-05T20:55:01.789 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-10-05T20:55:01.844 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-10-05T20:55:01.925 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-10-05T20:55:01.975 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-10-05T20:55:02.026 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-10-05T20:55:02.073 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-10-05T20:55:02.127 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-10-05T20:55:02.187 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-10-05T20:55:08.963 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-10-05T20:55:09.018 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-10-05T20:55:09.089 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-10-05T20:55:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-10-05T20:55:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-10-05T20:55:09.239 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-10-05T20:55:09.292 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-10-05T20:55:09.338 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-10-05T20:55:09.391 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-10-05T20:55:09.446 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-10-05T20:55:09.492 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-10-05T20:55:09.538 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-10-05T20:55:09.585 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-10-05T20:55:09.640 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-10-05T20:55:09.696 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-10-05T20:55:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-10-05T20:55:09.808 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-10-05T20:55:09.860 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T20:55:09.914 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T20:55:09.997 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T20:55:10.050 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T20:55:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T20:55:10.160 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T20:55:10.258 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-10-05T20:55:10.357 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-10-05T20:55:10.408 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-10-05T20:55:10.467 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-10-05T20:55:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-10-05T20:55:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-10-05T20:55:10.810 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-10-05T20:55:10.849 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-10-05T20:55:11.245 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-10-05T20:55:11.651 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-10-05T20:55:11.710 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-10-05T20:55:11.764 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-10-05T20:55:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-10-05T20:55:11.877 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-10-05T20:55:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-10-05T20:55:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-10-05T20:55:12.136 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-10-05T20:55:12.193 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-10-05T20:55:12.248 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-10-05T20:55:12.303 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-10-05T20:55:12.357 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-10-05T20:55:12.415 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-10-05T20:55:12.646 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-10-05T20:55:12.980 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-10-05T20:55:13.020 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-10-05T20:55:13.055 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-10-05T20:55:13.080 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-10-05T20:55:13.106 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-10-05T20:55:13.116 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-10-05T20:55:13.125 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-10-05T20:55:13.133 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-10-05T20:55:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-10-05T20:55:13.328 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-10-05T20:55:13.337 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-10-05T20:55:13.439 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-10-05T20:55:13.468 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-10-05T20:55:13.495 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-10-05T20:55:13.716 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-10-05T20:55:13.950 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-10-05T20:55:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-10-05T20:55:28.026 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-10-05T20:55:28.057 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-10-05T20:55:28.087 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-10-05T20:55:28.126 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-10-05T20:55:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-10-05T20:55:28.284 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-10-05T20:55:28.315 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-10-05T20:55:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-10-05T20:55:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-10-05T20:55:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-10-05T20:55:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-10-05T20:55:28.587 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-10-05T20:55:28.618 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-10-05T20:55:28.689 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-10-05T20:55:28.712 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-10-05T20:55:28.739 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-10-05T20:55:28.765 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-10-05T20:55:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-10-05T20:55:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-10-05T20:55:28.882 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-10-05T20:55:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-10-05T20:55:28.938 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-10-05T20:55:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-10-05T20:55:29.060 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-10-05T20:55:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-10-05T20:55:29.110 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-10-05T20:55:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-10-05T20:55:29.158 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-10-05T20:55:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-10-05T20:55:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-10-05T20:55:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-10-05T20:55:29.267 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-10-05T20:55:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-10-05T20:55:29.312 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-10-05T20:55:29.337 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-10-05T20:55:29.361 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-10-05T20:55:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-10-05T20:55:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-10-05T20:55:29.440 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-10-05T20:55:29.468 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-10-05T20:55:29.494 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-10-05T20:55:29.521 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-10-05T20:55:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-10-05T20:55:29.668 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-10-05T20:55:29.770 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-10-05T20:55:29.779 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-10-05T20:55:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-10-05T20:55:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-10-05T20:55:30.982 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-10-05T20:55:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-10-05T20:55:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-10-05T20:55:31.345 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-10-05T20:55:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-10-05T20:55:31.589 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-10-05T20:55:31.713 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-10-05T20:55:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-10-05T20:55:31.974 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-10-05T20:55:32.134 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-10-05T20:55:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-10-05T20:55:32.243 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-10-05T20:55:32.318 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-10-05T20:55:32.359 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-10-05T20:55:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-10-05T20:55:32.391 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-10-05T20:55:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-10-05T20:55:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-10-05T20:55:32.519 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-10-05T20:55:32.552 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-10-05T20:55:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-10-05T20:55:32.617 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-10-05T20:55:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-10-05T20:55:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-10-05T20:55:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-10-05T20:55:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-10-05T20:55:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-10-05T20:55:32.941 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-10-05T20:55:32.974 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-10-05T20:55:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-10-05T20:55:33.056 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-10-05T20:55:33.112 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-10-05T20:55:33.146 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-10-05T20:55:33.155 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-10-05T20:55:33.166 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-10-05T20:55:33.176 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-10-05T20:55:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-10-05T20:55:33.212 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-10-05T20:55:33.275 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-10-05T20:55:33.311 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-10-05T20:55:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-10-05T20:55:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-10-05T20:55:33.482 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-10-05T20:55:33.546 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-10-05T20:55:33.658 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-10-05T20:55:33.719 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-10-05T20:55:33.781 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-10-05T20:55:33.858 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-10-05T20:55:33.938 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-10-05T20:55:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-10-05T20:55:34.137 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-10-05T20:55:34.208 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-10-05T20:55:34.332 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-10-05T20:55:34.455 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-10-05T20:55:34.548 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-10-05T20:55:34.598 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-10-05T20:55:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-10-05T20:55:34.747 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-10-05T20:55:34.870 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-10-05T20:55:34.994 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-10-05T20:55:35.119 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-10-05T20:55:35.248 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-10-05T20:55:35.386 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-10-05T20:55:35.520 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-10-05T20:55:35.630 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-10-05T20:55:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-10-05T20:55:35.859 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-10-05T20:55:35.894 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-10-05T20:55:36.002 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-10-05T20:55:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-10-05T20:55:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-10-05T20:55:36.227 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-10-05T20:55:36.271 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-10-05T20:55:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-10-05T20:55:36.609 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-10-05T20:55:36.839 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-10-05T20:55:37.045 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-10-05T20:55:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-10-05T20:55:37.461 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-10-05T20:55:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-10-05T20:55:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-10-05T20:55:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-10-05T20:55:38.298 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-10-05T20:55:38.466 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-10-05T20:55:38.634 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-10-05T20:55:38.834 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-10-05T20:55:39.022 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-10-05T20:55:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-10-05T20:55:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-10-05T20:55:39.180 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-10-05T20:55:39.189 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-10-05T20:55:39.354 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-10-05T20:55:39.515 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-10-05T20:55:39.532 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-10-05T20:55:39.553 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-10-05T20:55:39.623 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-10-05T20:55:40.291 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-10-05T20:55:40.364 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-10-05T20:55:40.466 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-10-05T20:55:40.600 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-10-05T20:55:40.729 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-10-05T20:55:40.813 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-10-05T20:55:40.894 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-10-05T20:55:41.066 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-10-05T20:55:41.121 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-10-05T20:55:41.240 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-10-05T20:55:41.381 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-10-05T20:55:41.612 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-10-05T20:55:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-10-05T20:55:41.825 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-10-05T20:55:41.853 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-10-05T20:55:42.433 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-10-05T20:55:42.583 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-10-05T20:55:43.638 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-10-05T20:55:43.675 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-10-05T20:55:43.739 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-10-05T20:55:44.011 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-10-05T20:55:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-10-05T20:55:44.352 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-10-05T20:55:45.308 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-10-05T20:55:46.068 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-10-05T20:55:47.608 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-10-05T20:55:51.882 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-10-05T20:55:52.715 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-10-05T20:55:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-10-05T20:55:54.258 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-10-05T20:55:54.602 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-10-05T20:55:54.946 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-10-05T20:55:55.129 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-10-05T20:55:55.204 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-10-05T20:55:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-10-05T20:55:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-10-05T20:55:55.770 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-10-05T20:55:55.854 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-10-05T20:55:55.878 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-10-05T20:55:56.001 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-10-05T20:55:59.125 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-10-05T20:56:02.171 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-10-05T20:56:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-10-05T20:56:05.357 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-10-05T20:56:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-10-05T20:56:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-10-05T20:56:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-10-05T20:56:06.074 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-10-05T20:56:06.315 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-10-05T20:56:06.726 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-10-05T20:56:06.851 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-10-05T20:56:07.117 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-10-05T20:56:07.570 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-10-05T20:56:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-10-05T20:56:07.868 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-10-05T20:56:08.138 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-10-05T20:56:08.214 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-10-05T20:56:13.929 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-10-05T20:56:14.002 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-10-05T20:56:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-10-05T20:56:14.157 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-10-05T20:56:14.233 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-10-05T20:56:14.472 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-10-05T20:56:20.376 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-10-05T20:56:20.666 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-10-05T20:56:20.884 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-10-05T20:56:21.226 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-10-05T20:56:21.566 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-10-05T20:56:23.965 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-10-05T20:56:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-10-05T20:56:39.616 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-10-05T20:56:41.106 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-10-05T20:56:41.812 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-10-05T20:56:42.617 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-10-05T20:56:42.904 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-10-05T20:56:43.233 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-10-05T20:56:43.370 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-10-05T20:56:43.648 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-10-05T20:56:43.851 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-10-05T20:56:44.379 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-10-05T20:56:44.417 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-10-05T20:56:44.502 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-10-05T20:56:44.585 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-10-05T20:56:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-10-05T20:56:44.640 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-10-05T20:56:44.757 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-10-05T20:56:44.799 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-10-05T20:56:44.835 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-10-05T20:56:44.897 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-10-05T20:56:44.939 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-10-05T20:56:44.971 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-10-05T20:56:45.005 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-10-05T20:56:45.044 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-10-05T20:56:45.080 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-10-05T20:56:45.080 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-10-05T20:56:45.080 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-10-05T20:56:45.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-10-05T20:56:45.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-10-05T20:56:45.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-10-05T20:56:45.081 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-10-05T20:56:45.162 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-10-05T20:56:45.247 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-10-05T20:56:45.303 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-10-05T20:56:45.380 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-10-05T20:56:45.425 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-10-05T20:56:45.501 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-10-05T20:56:45.587 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-10-05T20:56:45.662 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-10-05T20:56:45.731 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-10-05T20:56:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-10-05T20:56:45.877 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-10-05T20:56:46.808 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-10-05T20:56:47.398 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-10-05T20:56:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-10-05T20:56:47.479 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-10-05T20:56:47.565 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-10-05T20:56:47.633 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-10-05T20:56:47.664 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-10-05T20:56:48.550 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-10-05T20:56:49.091 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-10-05T20:56:49.627 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-10-05T20:56:49.663 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-10-05T20:56:49.747 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-10-05T20:56:49.828 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-10-05T20:56:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-10-05T20:56:49.958 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-10-05T20:56:50.037 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-10-05T20:56:50.085 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-10-05T20:56:50.131 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-10-05T20:56:50.190 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-10-05T20:56:50.249 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-10-05T20:56:50.376 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-10-05T20:56:50.528 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-10-05T20:56:50.716 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-10-05T20:56:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-10-05T20:56:51.026 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-10-05T20:56:51.195 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-10-05T20:56:51.374 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-10-05T20:56:51.546 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-10-05T20:56:51.720 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-10-05T20:56:51.955 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-10-05T20:56:52.176 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-10-05T20:56:52.291 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-10-05T20:56:52.337 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-10-05T20:56:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-10-05T20:56:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-10-05T20:56:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-10-05T20:56:52.974 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-10-05T20:56:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-10-05T20:56:55.001 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-10-05T20:56:55.165 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-10-05T20:56:55.388 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-10-05T20:56:55.568 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-10-05T20:56:55.858 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-10-05T20:56:56.033 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-10-05T20:56:56.122 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-10-05T20:56:56.291 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-10-05T20:56:56.321 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-10-05T20:56:56.354 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-10-05T20:56:56.383 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-10-05T20:56:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-10-05T20:56:56.495 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-10-05T20:56:56.571 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-10-05T20:56:56.790 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-10-05T20:56:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-10-05T20:56:56.933 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-10-05T20:56:57.030 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-10-05T20:56:57.097 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-10-05T20:56:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-10-05T20:56:57.279 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-10-05T20:56:57.354 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-10-05T20:56:57.423 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-10-05T20:56:57.503 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-10-05T20:56:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-10-05T20:56:57.634 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-10-05T20:56:57.724 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-10-05T20:56:57.859 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-10-05T20:56:57.981 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-10-05T20:56:58.043 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-10-05T20:56:58.116 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-10-05T20:56:58.242 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-10-05T20:56:58.353 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-10-05T20:56:58.462 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-10-05T20:56:58.574 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-10-05T20:56:58.699 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-10-05T20:56:58.798 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-10-05T20:56:58.980 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-10-05T20:56:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-10-05T20:56:59.182 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-10-05T20:56:59.256 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-10-05T20:56:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-10-05T20:56:59.494 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-10-05T20:56:59.566 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-10-05T20:56:59.658 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-10-05T20:56:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-10-05T20:56:59.837 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-10-05T20:56:59.896 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-10-05T20:56:59.935 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-10-05T20:57:00.025 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-10-05T20:57:00.113 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-10-05T20:57:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-10-05T20:57:00.372 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-10-05T20:57:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-10-05T20:57:00.477 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-10-05T20:57:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-10-05T20:57:00.632 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-10-05T20:57:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-10-05T20:57:00.800 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-10-05T20:57:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-10-05T20:57:00.930 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-10-05T20:57:01.140 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-10-05T20:57:01.305 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-10-05T20:57:01.384 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-10-05T20:57:01.578 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-10-05T20:57:02.450 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-10-05T20:57:02.485 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-10-05T20:57:02.565 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-10-05T20:57:02.652 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-10-05T20:57:02.731 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-10-05T20:57:03.668 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-10-05T20:57:04.593 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-10-05T20:57:04.646 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-10-05T20:57:05.060 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-10-05T20:57:05.617 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-10-05T20:57:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-10-05T20:57:06.072 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-10-05T20:57:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-10-05T20:57:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-10-05T20:57:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-10-05T20:57:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-10-05T20:57:07.861 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-10-05T20:57:08.131 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-10-05T20:57:08.315 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-10-05T20:57:08.476 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-10-05T20:57:08.678 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-10-05T20:57:08.942 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-10-05T20:57:09.035 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-10-05T20:57:09.254 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-10-05T20:57:09.464 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-10-05T20:57:09.690 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-10-05T20:57:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-10-05T20:57:10.163 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-10-05T20:57:10.163 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-10-05T20:57:10.528 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-10-05T20:57:10.902 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-10-05T20:57:11.155 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-10-05T20:57:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-10-05T20:57:11.529 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-10-05T20:57:24.362 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-10-05T20:57:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr:---------------------------------------------------------------------- 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr:Ran 618 tests in 222.212s 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:58:24.740 INFO:teuthology.orchestra.run.smithi086.stderr:OK (SKIP=79) 2023-10-05T20:58:24.802 INFO:teuthology.orchestra.run.smithi086.stderr:Exception ignored in: .generate at 0x7f445ade7570> 2023-10-05T20:58:24.802 INFO:teuthology.orchestra.run.smithi086.stderr:RuntimeError: generator ignored GeneratorExit 2023-10-05T20:58:24.829 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-10-05T20:58:24.829 DEBUG:teuthology.orchestra.run.smithi086:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-10-05T20:58:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:58:24.898 INFO:tasks.s3tests:Cleaning up boto... 2023-10-05T20:58:24.898 DEBUG:teuthology.orchestra.run.smithi086:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T20:58:24.919 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-10-05T20:58:25.001 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:58:25.001 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:58:25.008 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.006+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 realm 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.010 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.008+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.009+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.009+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.009+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.011 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.009+0000 7f8fbe78a680 20 searching for the correct realm 2023-10-05T20:58:25.018 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.016+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.016+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.016+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.016+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.016+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.017+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.018+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.018+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.020 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.018+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.023+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.023+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.025 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.023+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.026 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.023+0000 7f8fbe78a680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.026 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.023+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.026 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.026 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.027 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.024+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 zone default found 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 4 Realm: () 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 zonegroup default 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.028 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 started sync module instance, tier type = 2023-10-05T20:58:25.029 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.025+0000 7f8fbe78a680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=0 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=1 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=2 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=3 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=4 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=5 2023-10-05T20:58:25.065 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.063+0000 7f8fbe78a680 20 add_watcher() i=6 2023-10-05T20:58:25.068 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.066+0000 7f8fbe78a680 20 add_watcher() i=7 2023-10-05T20:58:25.068 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.066+0000 7f8fbe78a680 2 all 8 watchers are set, enabling cache 2023-10-05T20:58:25.069 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.068+0000 7f8f837de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.069 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.068+0000 7f8fbe78a680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:58:25.070 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.068+0000 7f8fbe78a680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.128+0000 7f8ef27fc700 20 reqs_thread_entry: start 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8ee3fff700 20 reqs_thread_entry: start 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 20 init_complete bucket index max shards: 11 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8ee27fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:58:55 2023 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8ee1ffb700 20 reqs_thread_entry: start 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 20 get_system_obj_state: rctx=0x7ffd02e09778 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 20 Read xattr: user.rgw.idtag 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.129+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:58:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 20 get_system_obj_state: rctx=0x7ffd02e09430 obj=default.rgw.meta:root:.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.130+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 : miss 2023-10-05T20:58:25.135 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 info.flags=0x16 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 adding default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 to cache LRU end 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 20 Read xattr: user.rgw.acl 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 20 Read xattr: user.rgw.idtag 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=364 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 info.flags=0x13 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 moving default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 to cache LRU end 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.136 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 02A533 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.131+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got A74DE5 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 29DFE4 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 6EA965 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got A8769A 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 07B565 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 40DD9E 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 6998D9 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.132+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got A552FD 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 5150E5 2023-10-05T20:58:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 0A9388 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 4292DD 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 5AF02D 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got F5F3DB 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 3498F3 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.133+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got D592E2 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 331469 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 99BD51 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 9BFF34 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got AA9E0A 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got D16C01 2023-10-05T20:58:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got E9309A 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 75E3B5 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 55242F 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 962534 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got E2685C 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got F32B1C 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 756687 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.134+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got F94338 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:02A533 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 10 manifest: total_size = 38 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.9400066430650663355 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:02A533 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.135+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-10-05T20:58:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-10-05T20:58:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.138+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A74DE5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fbe78a680 10 manifest: total_size = 642384 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.15735982414042817873 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A74DE5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.0 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.139+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=02A533 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:29DFE4 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 10 manifest: total_size = 39911307 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.6744111194473048506 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:29DFE4 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.141+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.2 2023-10-05T20:58:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.142+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=A74DE5 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:58:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 RGWObjManifest::operator++(): result: ofs=39911307 stripe_ofs=39911307 part_ofs=0 rule->part_size=0 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - tag is: 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.6744111194473048506 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_1 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 332 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_2 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 564 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_3 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 796 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_4 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1028 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_5 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1260 2023-10-05T20:58:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_6 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1492 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_7 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1724 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_8 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1956 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518__shadow_.l6gm5aWtgs6hcn4Tf3K-y3JCEnBOGMd_9 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 2188 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-10-05T20:58:25.149 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.143+0000 7f8fbe78a680 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.6744111194473048506 2023-10-05T20:58:25.150 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.145+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:6EA965 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.145+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=29DFE4 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.146+0000 7f8fbe78a680 10 manifest: total_size = 399141 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.146+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.17345357502750170252 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.146+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:6EA965 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.146+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.146+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.2 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A8769A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.151 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 10 manifest: total_size = 399002 2023-10-05T20:58:25.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.11899199477715635868 2023-10-05T20:58:25.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A8769A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.148+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.2 2023-10-05T20:58:25.152 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.149+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=6EA965 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:07B565 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 10 manifest: total_size = 399226 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.9200056957297830960 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:07B565 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.153 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.151+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.3 2023-10-05T20:58:25.154 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.152+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=A8769A 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:40DD9E state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 10 manifest: total_size = 399108 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.9985574178500236933 2023-10-05T20:58:25.157 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:40DD9E state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.155+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.3 2023-10-05T20:58:25.158 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.156+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=07B565 2023-10-05T20:58:25.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.160 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.158+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:6998D9 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fbe78a680 10 manifest: total_size = 399230 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.3368372738113165939 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:6998D9 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.3 2023-10-05T20:58:25.161 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.159+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=40DD9E 2023-10-05T20:58:25.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.163 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A552FD state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 10 manifest: total_size = 180000 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.18438634684646198184 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:A552FD state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.161+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.3 2023-10-05T20:58:25.164 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.162+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=6998D9 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:5150E5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 10 manifest: total_size = 359315 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.5553259256415834395 2023-10-05T20:58:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:5150E5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.165+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.4 2023-10-05T20:58:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.166+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=A552FD 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:0A9388 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 10 manifest: total_size = 5 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.7350022584647869464 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:0A9388 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.168+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.5 2023-10-05T20:58:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.170+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=5150E5 2023-10-05T20:58:25.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.175 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.173+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:4292DD state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.174+0000 7f8fbe78a680 10 manifest: total_size = 575368 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.174+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.14535524934093794570 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.174+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:4292DD state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.174+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.176 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.174+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.5 2023-10-05T20:58:25.177 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.176+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=0A9388 2023-10-05T20:58:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:5AF02D state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 10 manifest: total_size = 399192 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.12629117151971316666 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:5AF02D state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.180+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.5 2023-10-05T20:58:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.182+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=4292DD 2023-10-05T20:58:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F5F3DB state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 10 manifest: total_size = 5 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.15185140187595324702 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F5F3DB state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.187+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.5 2023-10-05T20:58:25.194 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.193+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=5AF02D 2023-10-05T20:58:25.198 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.196+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:3498F3 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8fbe78a680 10 manifest: total_size = 575092 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.14335676048677353869 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:3498F3 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.199 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.6 2023-10-05T20:58:25.200 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.197+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=F5F3DB 2023-10-05T20:58:25.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.203 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:D592E2 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 10 manifest: total_size = 399192 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.15206844321834985664 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:D592E2 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.204 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.201+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.6 2023-10-05T20:58:25.206 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.204+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=3498F3 2023-10-05T20:58:25.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.214 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:331469 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 10 manifest: total_size = 590142 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.9984887579655370161 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:331469 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.215 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.212+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.218 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.216+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=D592E2 2023-10-05T20:58:25.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.221 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.219+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:99BD51 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.220+0000 7f8fbe78a680 10 manifest: total_size = 399044 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.220+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.2104385707161651937 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.220+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:99BD51 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.220+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.222 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.220+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.224 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.222+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=331469 2023-10-05T20:58:25.229 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:9BFF34 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 10 manifest: total_size = 399119 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.4470319899033602289 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:9BFF34 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.228+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.232 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.230+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=99BD51 2023-10-05T20:58:25.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.238 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:AA9E0A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 10 manifest: total_size = 399046 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.8812100037449804711 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:AA9E0A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.239 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.236+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.240 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.239+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=9BFF34 2023-10-05T20:58:25.246 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.244+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:D16C01 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.245+0000 7f8fbe78a680 10 manifest: total_size = 399156 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.245+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.3837319261680332842 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.245+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:D16C01 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.245+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.247 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.245+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.249 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.247+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=AA9E0A 2023-10-05T20:58:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.252+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:E9309A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.253+0000 7f8fbe78a680 10 manifest: total_size = 399098 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.253+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.2583491432597452292 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.253+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:E9309A state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.253+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.253+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.7 2023-10-05T20:58:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.257+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=D16C01 2023-10-05T20:58:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:75E3B5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 10 manifest: total_size = 117 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.1809663668399077083 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:75E3B5 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.261+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.8 2023-10-05T20:58:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.264+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=E9309A 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:55242F state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 10 manifest: total_size = 49280 2023-10-05T20:58:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.13510239085434207571 2023-10-05T20:58:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:55242F state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.269+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.9 2023-10-05T20:58:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.270+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=75E3B5 2023-10-05T20:58:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:962534 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 10 manifest: total_size = 180000 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.5273034119179668888 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:962534 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.272+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.9 2023-10-05T20:58:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.273+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=55242F 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:E2685C state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 10 manifest: total_size = 643408 2023-10-05T20:58:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.3178142423473782781 2023-10-05T20:58:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:E2685C state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.275+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.9 2023-10-05T20:58:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.276+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=962534 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F32B1C state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 10 manifest: total_size = 398980 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.6194168302560313999 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F32B1C state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.278+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.9 2023-10-05T20:58:25.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.280+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=E2685C 2023-10-05T20:58:25.287 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.285+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:756687 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.286+0000 7f8fbe78a680 10 manifest: total_size = 399218 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.286+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.5509127190088782027 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.286+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:756687 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.286+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.288 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.286+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.10 2023-10-05T20:58:25.290 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.288+0000 7f8fab7fe700 20 handle_completion(): completion ok for obj=F32B1C 2023-10-05T20:58:25.295 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.295 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.293+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F94338 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.294+0000 7f8fbe78a680 10 manifest: total_size = 399220 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.294+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.15034320534981744630 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.294+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=test:F94338 state=0x55a723028338 s->prefetch_data=0 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.294+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a723028338 2023-10-05T20:58:25.296 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.294+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518.10 2023-10-05T20:58:25.298 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.296+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=756687 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T20:58:25.302 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T20:58:25.303 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.300+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.302+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=F94338 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518])"; uploads.size()=0; is_truncated=0 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]) 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 lifecycle: get_system_obj_state: rctx=0x7ffd02e08f08 obj=default.rgw.meta:root:test state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T20:58:25.305 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-10-05T20:58:25.306 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.303+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 info.flags=0x17 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8fbe78a680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 to cache LRU end 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.307 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8fbe78a680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518, ofs0, ns] 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856238 cookie 94176334165920 notifier 4243 bl.length()=800 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8faaffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 info.flags=0x17 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8faaffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 to cache LRU end 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.305+0000 7f8faaffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.306+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]): orig_sources= new_sources= 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.306+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]): potential sources added=[] removed=[] 2023-10-05T20:58:25.308 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.306+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]): orig_dests= new_dests= 2023-10-05T20:58:25.309 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.306+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:test[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518]): potential dests added=[] removed=[] 2023-10-05T20:58:25.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 20 get_system_obj_state: rctx=0x7ffd02e08968 obj=default.rgw.meta:root:test state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.313 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 moving default.rgw.meta+root+test to cache LRU end 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-10-05T20:58:25.314 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 removing default.rgw.meta+root+test from cache 2023-10-05T20:58:25.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-10-05T20:58:25.315 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.312+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856239 cookie 94176334179408 notifier 4243 bl.length()=148 2023-10-05T20:58:25.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.314+0000 7f8fbe78a680 10 removing default.rgw.meta+root+.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 from cache 2023-10-05T20:58:25.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.314+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518, ofs0, ns] 2023-10-05T20:58:25.316 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.314+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856240 cookie 94176334165920 notifier 4243 bl.length()=207 2023-10-05T20:58:25.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.322+0000 7f8fbe78a680 10 removing default.rgw.log++pubsub..bucket.test/7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518 from cache 2023-10-05T20:58:25.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.322+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/7aa773d0-e721-48b8-a101-2fb8f112c827.4203.518, ofs0, ns] 2023-10-05T20:58:25.324 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.323+0000 7f8fab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855635 cookie 94176334155872 notifier 4243 bl.length()=204 2023-10-05T20:58:25.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.325+0000 7f8fbe78a680 20 get_system_obj_state: rctx=0x7ffd02e09430 obj=default.rgw.meta:root:.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.327 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.325+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 : miss 2023-10-05T20:58:25.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 info.flags=0x16 2023-10-05T20:58:25.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 to cache LRU end 2023-10-05T20:58:25.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 20 Read xattr: user.rgw.acl 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 20 Read xattr: user.rgw.idtag 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 info.flags=0x13 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 to cache LRU end 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 2023-10-05T20:58:25.329 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.326+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T20:58:25.330 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.327+0000 7f8fbe78a680 10 cls_bucket_list_unordered: got 70638B 2023-10-05T20:58:25.331 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.329+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=testbuck2:70638B state=0x55a722eddf08 s->prefetch_data=0 2023-10-05T20:58:25.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.330+0000 7f8fbe78a680 10 manifest: total_size = 399119 2023-10-05T20:58:25.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.330+0000 7f8fbe78a680 20 get_obj_state: setting s->obj_tag to 7aa773d0-e721-48b8-a101-2fb8f112c827.4201.9774727337668183246 2023-10-05T20:58:25.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.330+0000 7f8fbe78a680 20 get_obj_state: rctx=0x7ffd02e09440 obj=testbuck2:70638B state=0x55a722eddf08 s->prefetch_data=0 2023-10-05T20:58:25.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.330+0000 7f8fbe78a680 20 prepare_atomic_modification: state is not atomic. state=0x55a722eddf08 2023-10-05T20:58:25.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.330+0000 7f8fbe78a680 20 bucket index object: default.rgw.buckets.index:.dir.7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519.1 2023-10-05T20:58:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.332+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-10-05T20:58:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.332+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-10-05T20:58:25.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-10-05T20:58:25.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-10-05T20:58:25.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 chain_cache_entry: cache_locator= 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T20:58:25.336 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.333+0000 7f8fbe78a680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8faaffd700 20 handle_completion(): completion ok for obj=70638B 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519])"; uploads.size()=0; is_truncated=0 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]) 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 lifecycle: get_system_obj_state: rctx=0x7ffd02e08f08 obj=default.rgw.meta:root:testbuck2 state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.337 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-10-05T20:58:25.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T20:58:25.338 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.334+0000 7f8fbe78a680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.335+0000 7f8fbe78a680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.335+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.335+0000 7f8fbe78a680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.335+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 info.flags=0x17 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 to cache LRU end 2023-10-05T20:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 10 lifecycle: distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519, ofs0, ns] 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856241 cookie 94176334165920 notifier 4243 bl.length()=810 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8faaffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 info.flags=0x17 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8faaffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 to cache LRU end 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8faaffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]): orig_sources= new_sources= 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]): potential sources added=[] removed=[] 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]): orig_dests= new_dests= 2023-10-05T20:58:25.340 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.336+0000 7f8fbe78a680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519]): potential dests added=[] removed=[] 2023-10-05T20:58:25.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.343+0000 7f8fbe78a680 20 get_system_obj_state: rctx=0x7ffd02e08968 obj=default.rgw.meta:root:testbuck2 state=0x55a722d91240 s->prefetch_data=0 2023-10-05T20:58:25.345 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.343+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-10-05T20:58:25.346 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.344+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856242 cookie 94176334173152 notifier 4243 bl.length()=153 2023-10-05T20:58:25.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.346+0000 7f8fbe78a680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 from cache 2023-10-05T20:58:25.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.346+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519, ofs0, ns] 2023-10-05T20:58:25.348 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.346+0000 7f8fab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856243 cookie 94176334165920 notifier 4243 bl.length()=212 2023-10-05T20:58:25.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.354+0000 7f8fbe78a680 10 removing default.rgw.log++pubsub..bucket.testbuck2/7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519 from cache 2023-10-05T20:58:25.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.354+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/7aa773d0-e721-48b8-a101-2fb8f112c827.4203.519, ofs0, ns] 2023-10-05T20:58:25.356 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.355+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855636 cookie 94176334155872 notifier 4243 bl.length()=209 2023-10-05T20:58:25.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.358+0000 7f8fbe78a680 10 removing key index: KTOYILKNEKKGRDDHYUJN 2023-10-05T20:58:25.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.359+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:KTOYILKNEKKGRDDHYUJN, ofs0, ns] 2023-10-05T20:58:25.360 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.359+0000 7f8fab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855637 cookie 94176334155872 notifier 4243 bl.length()=170 2023-10-05T20:58:25.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.360+0000 7f8fbe78a680 10 removing email index: foo.client.0+test@test.test 2023-10-05T20:58:25.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.360+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T20:58:25.362 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.360+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856244 cookie 94176334179408 notifier 4243 bl.length()=178 2023-10-05T20:58:25.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.362+0000 7f8fbe78a680 10 removing user buckets index 2023-10-05T20:58:25.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.362+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-10-05T20:58:25.364 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.362+0000 7f8fab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856245 cookie 94176334165920 notifier 4243 bl.length()=169 2023-10-05T20:58:25.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.364+0000 7f8fbe78a680 10 removing user index: foo.client.0 2023-10-05T20:58:25.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.364+0000 7f8fbe78a680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-10-05T20:58:25.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.364+0000 7f8fbe78a680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T20:58:25.366 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.365+0000 7f8faaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855638 cookie 94176334155872 notifier 4243 bl.length()=161 2023-10-05T20:58:25.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.367+0000 7f8fbe78a680 20 remove_watcher() i=0 2023-10-05T20:58:25.368 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.367+0000 7f8fbe78a680 2 removed watcher, disabling cache 2023-10-05T20:58:25.369 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.367+0000 7f8fbe78a680 20 remove_watcher() i=1 2023-10-05T20:58:25.371 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.370+0000 7f8fbe78a680 20 remove_watcher() i=2 2023-10-05T20:58:25.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.371+0000 7f8fbe78a680 20 remove_watcher() i=3 2023-10-05T20:58:25.373 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.372+0000 7f8fbe78a680 20 remove_watcher() i=4 2023-10-05T20:58:25.374 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.373+0000 7f8fbe78a680 20 remove_watcher() i=5 2023-10-05T20:58:25.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.375+0000 7f8fbe78a680 20 remove_watcher() i=6 2023-10-05T20:58:25.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.378+0000 7f8fbe78a680 20 remove_watcher() i=7 2023-10-05T20:58:25.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.378+0000 7f8f837de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.386 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-10-05T20:58:25.469 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:58:25.470 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:58:25.475 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.474+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 realm 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.477 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.475+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.476+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.478 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.476+0000 7f6af7fd6680 20 searching for the correct realm 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.482+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.482+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.482+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.482+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.482+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.486 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.483+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.487 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.484+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.489+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.489+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.489+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.489+0000 7f6af7fd6680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.489+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.492 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.490+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 zone default found 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 4 Realm: () 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 zonegroup default 2023-10-05T20:58:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 started sync module instance, tier type = 2023-10-05T20:58:25.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.491+0000 7f6af7fd6680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:58:25.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=0 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=1 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=2 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=3 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=4 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=5 2023-10-05T20:58:25.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.513+0000 7f6af7fd6680 20 add_watcher() i=6 2023-10-05T20:58:25.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.515+0000 7f6af7fd6680 20 add_watcher() i=7 2023-10-05T20:58:25.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.515+0000 7f6af7fd6680 2 all 8 watchers are set, enabling cache 2023-10-05T20:58:25.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.518+0000 7f6a37fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.518+0000 7f6af7fd6680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:58:25.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.518+0000 7f6af7fd6680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:58:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.604+0000 7f6a26ffd700 20 reqs_thread_entry: start 2023-10-05T20:58:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.604+0000 7f6a257fa700 20 reqs_thread_entry: start 2023-10-05T20:58:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 20 init_complete bucket index max shards: 11 2023-10-05T20:58:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:58:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6a1f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:58:56 2023 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6a1effd700 20 reqs_thread_entry: start 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 20 get_system_obj_state: rctx=0x7ffdb5b269e8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x5639a1fef240 s->prefetch_data=0 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 20 Read xattr: user.rgw.idtag 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.605+0000 7f6af7fd6680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T20:58:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 10 removing key index: TGPNIQLUVGCFOOTSCTXL 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6af7fd6680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:TGPNIQLUVGCFOOTSCTXL, ofs0, ns] 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.606+0000 7f6ae90aa700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856246 cookie 94805532588816 notifier 4249 bl.length()=170 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.607+0000 7f6af7fd6680 10 removing email index: bar.client.0+test@test.test 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.607+0000 7f6af7fd6680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T20:58:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.608+0000 7f6ae88a9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856247 cookie 94805532576016 notifier 4249 bl.length()=178 2023-10-05T20:58:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.609+0000 7f6af7fd6680 10 removing user buckets index 2023-10-05T20:58:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.609+0000 7f6af7fd6680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-10-05T20:58:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.609+0000 7f6ae90aa700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323857493 cookie 94805532573712 notifier 4249 bl.length()=169 2023-10-05T20:58:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.610+0000 7f6af7fd6680 10 removing user index: bar.client.0 2023-10-05T20:58:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.610+0000 7f6af7fd6680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-10-05T20:58:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.610+0000 7f6af7fd6680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T20:58:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.611+0000 7f6ae88a9700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856248 cookie 94805532576016 notifier 4249 bl.length()=161 2023-10-05T20:58:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.613+0000 7f6af7fd6680 20 remove_watcher() i=0 2023-10-05T20:58:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.613+0000 7f6af7fd6680 2 removed watcher, disabling cache 2023-10-05T20:58:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.614+0000 7f6af7fd6680 20 remove_watcher() i=1 2023-10-05T20:58:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.615+0000 7f6af7fd6680 20 remove_watcher() i=2 2023-10-05T20:58:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.615+0000 7f6af7fd6680 20 remove_watcher() i=3 2023-10-05T20:58:25.618 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.616+0000 7f6af7fd6680 20 remove_watcher() i=4 2023-10-05T20:58:25.619 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.617+0000 7f6af7fd6680 20 remove_watcher() i=5 2023-10-05T20:58:25.620 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.619+0000 7f6af7fd6680 20 remove_watcher() i=6 2023-10-05T20:58:25.621 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.620+0000 7f6af7fd6680 20 remove_watcher() i=7 2023-10-05T20:58:25.622 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.620+0000 7f6a37fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.628 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-10-05T20:58:25.670 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:58:25.670 INFO:teuthology.orchestra.run.smithi086.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:58:25.676 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.674+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.677 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.675+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.675+0000 7fdb85f34680 20 realm 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.675+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.678 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.676+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.677+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.679 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.677+0000 7fdb85f34680 20 searching for the correct realm 2023-10-05T20:58:25.685 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.683+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.686 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.684+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.685+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.690+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zone_info.7aa773d0-e721-48b8-a101-2fb8f112c827 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.690+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.690+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.690+0000 7fdb85f34680 20 RGWRados::pool_iterate: got zonegroup_info.4053bbbb-08de-4e53-942b-485ef7605f41 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.690+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:58:25.693 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.691+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 zone default found 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 4 Realm: () 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 4 ZoneGroup: default (4053bbbb-08de-4e53-942b-485ef7605f41) 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 4 Zone: default (7aa773d0-e721-48b8-a101-2fb8f112c827) 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:58:25.695 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 zonegroup default 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 started sync module instance, tier type = 2023-10-05T20:58:25.696 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.692+0000 7fdb85f34680 20 started zone id=7aa773d0-e721-48b8-a101-2fb8f112c827 (name=default) with tier type = 2023-10-05T20:58:25.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=0 2023-10-05T20:58:25.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=1 2023-10-05T20:58:25.708 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=2 2023-10-05T20:58:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=3 2023-10-05T20:58:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=4 2023-10-05T20:58:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=5 2023-10-05T20:58:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.707+0000 7fdb85f34680 20 add_watcher() i=6 2023-10-05T20:58:25.711 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.709+0000 7fdb85f34680 20 add_watcher() i=7 2023-10-05T20:58:25.711 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.709+0000 7fdb85f34680 2 all 8 watchers are set, enabling cache 2023-10-05T20:58:25.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.712+0000 7fdb4a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.712+0000 7fdb85f34680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:58:25.714 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.712+0000 7fdb85f34680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdab97fa700 20 reqs_thread_entry: start 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdab2ffd700 20 reqs_thread_entry: start 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdb85f34680 20 init_complete bucket index max shards: 11 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdb85f34680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdab17fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:58:55 2023 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdab0ff9700 20 reqs_thread_entry: start 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdb85f34680 20 get_system_obj_state: rctx=0x7fffa2535638 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55769de51240 s->prefetch_data=0 2023-10-05T20:58:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.764+0000 7fdb85f34680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 20 Read xattr: user.rgw.idtag 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 20 rados->read ofs=0 len=0 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 20 rados_obj.operate() r=0 bl.length=528 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:58:25.768 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:58:25.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 removing key index: HPMVCDIDZBCEPDLRLFZJ 2023-10-05T20:58:25.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.765+0000 7fdb85f34680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:HPMVCDIDZBCEPDLRLFZJ, ofs0, ns] 2023-10-05T20:58:25.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.766+0000 7fdb6affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856249 cookie 93967945153904 notifier 4255 bl.length()=170 2023-10-05T20:58:25.769 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.767+0000 7fdb85f34680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-10-05T20:58:25.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.767+0000 7fdb85f34680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T20:58:25.770 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.767+0000 7fdb6a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855639 cookie 93967945172704 notifier 4255 bl.length()=193 2023-10-05T20:58:25.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.770+0000 7fdb85f34680 10 removing user buckets index 2023-10-05T20:58:25.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.770+0000 7fdb85f34680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-10-05T20:58:25.771 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.770+0000 7fdb6affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856250 cookie 93967945161248 notifier 4255 bl.length()=184 2023-10-05T20:58:25.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.771+0000 7fdb85f34680 10 removing user index: testx$tenanteduser.client.0 2023-10-05T20:58:25.773 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.771+0000 7fdb85f34680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-10-05T20:58:25.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.771+0000 7fdb85f34680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T20:58:25.774 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.772+0000 7fdb6a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855640 cookie 93967945166560 notifier 4255 bl.length()=176 2023-10-05T20:58:25.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.774+0000 7fdb85f34680 20 remove_watcher() i=0 2023-10-05T20:58:25.776 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.774+0000 7fdb85f34680 2 removed watcher, disabling cache 2023-10-05T20:58:25.779 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.777+0000 7fdb85f34680 20 remove_watcher() i=1 2023-10-05T20:58:25.782 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.781+0000 7fdb85f34680 20 remove_watcher() i=2 2023-10-05T20:58:25.785 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.784+0000 7fdb85f34680 20 remove_watcher() i=3 2023-10-05T20:58:25.788 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.786+0000 7fdb85f34680 20 remove_watcher() i=4 2023-10-05T20:58:25.791 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.789+0000 7fdb85f34680 20 remove_watcher() i=5 2023-10-05T20:58:25.793 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.792+0000 7fdb85f34680 20 remove_watcher() i=6 2023-10-05T20:58:25.796 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.795+0000 7fdb85f34680 20 remove_watcher() i=7 2023-10-05T20:58:25.797 INFO:teuthology.orchestra.run.smithi086.stderr:2023-10-05T20:58:25.795+0000 7fdb4a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:58:25.802 INFO:tasks.s3tests:Removing s3-tests... 2023-10-05T20:58:25.802 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T20:58:25.965 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-10-05T20:58:25.975 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T20:58:25.984 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T20:58:25.984 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:25.985 INFO:tasks.rgw.client.0.smithi086.stdout:2023-10-05T20:58:25.983+0000 7f1ab3916700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 119227) UID: 0 2023-10-05T20:58:25.985 INFO:tasks.rgw.client.0.smithi086.stdout:2023-10-05T20:58:25.983+0000 7f1ac3ca5740 -1 shutting down 2023-10-05T20:58:27.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T20:58:32.087 INFO:tasks.rgw.client.0:Stopped 2023-10-05T20:58:32.087 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T20:58:32.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T20:58:32.183 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T20:58:32.192 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T20:58:32.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T20:58:32.543 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-10-05T20:58:32.544 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":35,"fsid":"d60c04eb-231a-4399-a9de-74389c06e19a","created":"2023-10-05T20:48:29.141562+0000","modified":"2023-10-05T20:55:36.249497+0000","last_up_change":"2023-10-05T20:48:36.875323+0000","last_in_change":"2023-10-05T20:48:30.118607+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-05T20:48:37.849330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-05T20:48:38.858982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T20:48:43.338734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T20:48:45.323648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T20:48:47.201873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T20:48:49.041187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T20:48:51.023707+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T20:48:53.080252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-10-05T20:49:03.216727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-10-05T20:54:43.777573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"6e7917e6-a209-45f7-b03c-855e6ddb30b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":114418},{"type":"v1","addr":"172.21.15.86:6801","nonce":114418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":114418},{"type":"v1","addr":"172.21.15.86:6803","nonce":114418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":114418},{"type":"v1","addr":"172.21.15.86:6807","nonce":114418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":114418},{"type":"v1","addr":"172.21.15.86:6805","nonce":114418}]},"public_addr":"172.21.15.86:6801/114418","cluster_addr":"172.21.15.86:6803/114418","heartbeat_back_addr":"172.21.15.86:6807/114418","heartbeat_front_addr":"172.21.15.86:6805/114418","state":["exists","up"]},{"osd":1,"uuid":"7c3965a7-b09a-4fe4-9205-bc4c0a5028db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":114419},{"type":"v1","addr":"172.21.15.86:6809","nonce":114419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":114419},{"type":"v1","addr":"172.21.15.86:6811","nonce":114419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":114419},{"type":"v1","addr":"172.21.15.86:6815","nonce":114419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":114419},{"type":"v1","addr":"172.21.15.86:6813","nonce":114419}]},"public_addr":"172.21.15.86:6809/114419","cluster_addr":"172.21.15.86:6811/114419","heartbeat_back_addr":"172.21.15.86:6815/114419","heartbeat_front_addr":"172.21.15.86:6813/114419","state":["exists","up"]},{"osd":2,"uuid":"1171759f-5db6-4d73-bf0e-d66423f5cbc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":114448},{"type":"v1","addr":"172.21.15.86:6817","nonce":114448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":114448},{"type":"v1","addr":"172.21.15.86:6819","nonce":114448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":114448},{"type":"v1","addr":"172.21.15.86:6823","nonce":114448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":114448},{"type":"v1","addr":"172.21.15.86:6821","nonce":114448}]},"public_addr":"172.21.15.86:6817/114448","cluster_addr":"172.21.15.86:6819/114448","heartbeat_back_addr":"172.21.15.86:6823/114448","heartbeat_front_addr":"172.21.15.86:6821/114448","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.571288+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.641704+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T20:48:34.588127+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.39","mappings":[{"from":0,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"4.23","mappings":[{"from":1,"to":0}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"4.3f","mappings":[{"from":1,"to":0}]},{"pgid":"10.1a","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:58:32.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T20:58:32.880 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T20:58:32.880 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T20:58:32.880 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T20:58:32.881 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:32.931 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T20:58:32.932 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T20:58:32.932 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:32.989 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T20:58:32.990 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T20:58:32.990 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:33.034 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T20:58:33.034 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T20:58:33.035 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T20:58:33.035 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:33.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-05T20:58:33.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T20:58:33.057 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T20:58:33.057 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T20:58:33.058 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T20:58:33.058 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:58:33.109 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T20:58:33.109 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T20:58:33.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:58:33.177 INFO:teuthology.orchestra.run.smithi086.stdout:1696538918.9269347 mon.a (mon.0) 75 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T20:58:33.178 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T20:58:33.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:58:33.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:58:33.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:58:33.382 INFO:teuthology.orchestra.run.smithi086.stdout:1696538918.9269347 mon.a (mon.0) 75 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T20:58:33.382 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:58:33.382 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T20:58:35.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:58:35.627 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T20:58:35.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:58:35.704 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T20:58:35.772 INFO:tasks.ceph:Archiving mon data... 2023-10-05T20:58:35.774 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757/data/mon.a.tgz 2023-10-05T20:58:35.774 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-10-05T20:58:35.794 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.3Ka31PZnur 2023-10-05T20:58:35.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3Ka31PZnur 2023-10-05T20:58:35.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.3Ka31PZnur 2023-10-05T20:58:36.041 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.3Ka31PZnur is 212KB 2023-10-05T20:58:36.133 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.3Ka31PZnur 2023-10-05T20:58:36.150 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T20:58:36.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T20:58:36.216 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T20:58:36.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T20:58:36.284 INFO:teuthology.orchestra.run.smithi086.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T20:58:36.290 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T20:58:36.292 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757/remote/smithi086/crash 2023-10-05T20:58:36.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T20:58:36.357 INFO:tasks.ceph:Compressing logs... 2023-10-05T20:58:36.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T20:58:38.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-05T20:59:05.286 INFO:tasks.ceph:Archiving logs... 2023-10-05T20:59:05.286 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757/remote/smithi086/log 2023-10-05T20:59:05.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T20:59:08.560 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T20:59:08.570 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T20:59:08.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T20:59:08.632 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T20:59:08.642 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T20:59:08.642 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T20:59:08.663 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-10-05T20:59:08.667 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T20:59:08.667 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-10-05T20:59:08.668 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 109 -639us[ -639us] +/- 71ms 2023-10-05T20:59:08.668 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 9 377 116 -728us[ -731us] +/- 63ms 2023-10-05T20:59:08.668 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 9 377 185 +1309us[+1305us] +/- 84ms 2023-10-05T20:59:08.668 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:59:08.669 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T20:59:08.678 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T20:59:08.678 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T20:59:08.689 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-05T20:59:08.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T20:59:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:59:08.833 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T20:59:08.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T20:59:08.855 INFO:teuthology.task.internal:Duration was 4807.207335 seconds 2023-10-05T20:59:08.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T20:59:08.867 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T20:59:08.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T20:59:08.920 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T20:59:09.231 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T20:59:09.231 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2023-10-05T20:59:09.231 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T20:59:09.254 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T20:59:09.254 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T20:59:09.331 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T20:59:09.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T20:59:09.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T20:59:09.458 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T20:59:09.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T20:59:09.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T20:59:09.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T20:59:09.552 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2023-10-05T20:59:09.576 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T20:59:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:59:09.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T20:59:09.645 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T20:59:09.646 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757/remote/smithi086 2023-10-05T20:59:09.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T20:59:09.711 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T20:59:09.711 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T20:59:09.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T20:59:09.772 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T20:59:09.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T20:59:09.786 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T20:59:09.787 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T20:59:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: 525641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 20:59 /home/ubuntu/cephtest 2023-10-05T20:59:09.821 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T20:59:09.833 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T20:59:09.868 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T20:59:09.891 DEBUG:teuthology.nuke:shortname: smithi086 2023-10-05T20:59:09.892 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T20:59:09.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411757', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:20:58.448078', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqsJPXSyQ8/UoKTpMll+bKPWbkWz3arfKZaC6WX3RqXyZCaa/eZYDhqtXH65cBInWzNcuT84sUvnATNTf/gbwA='} 2023-10-05T20:59:09.963 INFO:teuthology.orchestra.console.smithi086:Power off 2023-10-05T20:59:09.963 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T20:59:09.985 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2023-10-05T20:59:09.985 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:59:10.008 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-10-05T20:59:14.009 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:59:14.118 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-10-05T20:59:18.119 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:59:18.228 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2023-10-05T20:59:22.230 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:59:22.339 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2023-10-05T20:59:22.440 INFO:teuthology.orchestra.console.smithi086:Power off completed 2023-10-05T20:59:22.544 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} duration: 4807.207334518433 failure_reason: '"1696538918.9269347 mon.a (mon.0) 75 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T20:59:22.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T20:59:22.616 INFO:teuthology.run:FAIL