2023-10-05T19:30:59.907 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:30:59.907 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761 --verbose --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761/orig.config.yaml 2023-10-05T19:30:59.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:31:00.015 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7411761' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt vault auth: token rgw crypt vault prefix: /v1/kv/data rgw crypt vault secret engine: kv setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 rgw: client.0: use-vault-role: client.0 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvnFQtyyPRYWPgri6uc34sDcJemY45OMhSqNvFCy5pir0u3AHWCvD56dzwRnNfAz+Y0TckFOlVUYp/kjtEG/EA= tasks: - install: null - ceph: null - vault: client.0: engine: kv install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/kv/data/ root_token: test_root_token secrets: - path: my-key-1 secret: a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c= - path: my-key-2 secret: a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg= - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:31:00.015 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:31:00.016 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:31:00.016 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:31:00.017 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:31:00.040 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:31:00.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:31:00.040 INFO:teuthology.packaging:ref: None 2023-10-05T19:31:00.040 INFO:teuthology.packaging:tag: None 2023-10-05T19:31:00.040 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:31:00.040 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:31:00.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:31:00.209 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278.g487ffcb5 2023-10-05T19:31:00.209 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:31:00.218 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:31:00.218 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:31:00.231 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:31:00.244 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:31:00.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:31:00.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:25:50.409662', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvnFQtyyPRYWPgri6uc34sDcJemY45OMhSqNvFCy5pir0u3AHWCvD56dzwRnNfAz+Y0TckFOlVUYp/kjtEG/EA='} 2023-10-05T19:31:00.281 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:31:00.292 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-05T19:31:00.292 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:31:00.340 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8d7023bee0>, signals=[15]) 2023-10-05T19:31:00.340 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:31:00.351 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:31:00.351 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:31:00.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:31:00.420 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:31:00.431 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-10-05T19:31:00.489 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-10-05T19:31:00.489 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.4 (Ootpa)" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.4" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-05T19:31:00.543 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-05T19:31:00.544 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-05T19:31:00.545 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-10-05T19:31:00.568 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:31:00.582 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:31:00.594 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:31:00.595 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:31:00.609 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:31:00.619 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:31:00.620 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:31:00.663 INFO:teuthology.orchestra.run.smithi172.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-05T19:31:00.664 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:31:00.699 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:31:00.717 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:31:00.717 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:31:00.718 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:31:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:31:00.731 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-05T19:31:00.785 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-05T19:31:00.785 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-305.25.1.el8_4.x86_64 2023-10-05T19:31:00.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2023-10-05T19:31:02.354 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:31:02.355 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-05T19:31:03.733 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 33 kB/s | 3.5 kB 00:00 2023-10-05T19:31:04.339 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 26 MB/s | 16 MB 00:00 2023-10-05T19:31:07.647 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-05T19:31:08.365 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-05T19:31:08.366 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-05T19:31:08.366 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-05T19:31:08.520 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:31:08.521 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:31:08.521 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:31:08.638 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2023-10-05T19:31:08.985 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:31:08.985 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-05T19:31:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Thu 05 Oct 2023 07:31:07 PM UTC. 2023-10-05T19:31:09.430 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-05T19:31:09.431 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:31:09.431 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:31:09.466 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-05T19:31:09.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2023-10-05T19:31:09.860 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:31:09.860 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-10-05T19:31:10.144 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:03 ago on Thu 05 Oct 2023 07:31:07 PM UTC. 2023-10-05T19:31:10.266 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-05T19:31:10.266 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:31:10.267 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:31:10.306 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-10-05T19:31:10.357 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:31:10.483 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.483 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.483 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:31:10.483 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:31:10.483 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:31:10.484 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:31:10.484 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:31:10.608 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.609 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2023-10-05T19:31:10.636 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2023-10-05T19:31:10.636 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2023-10-05T19:31:10.636 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.02 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 99.el8_4.1 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Group : System Environment/Base 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 9264631 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Relocations : (not relocatable) 2023-10-05T19:31:10.637 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : Red Hat, Inc. 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : Red Hat, Inc. 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:31:10.638 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2023-10-05T19:31:10.639 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-05T19:31:10.639 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:10.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-05T19:31:12.064 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2023-10-05T19:31:14.415 INFO:teuthology.orchestra.run.smithi172.stderr:done 2023-10-05T19:31:14.416 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-05T19:31:14.430 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.yBa1dwvpia 2023-10-05T19:31:14.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yBa1dwvpia 2023-10-05T19:31:14.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.yBa1dwvpia 2023-10-05T19:31:14.631 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.yBa1dwvpia is 6KB 2023-10-05T19:31:14.679 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.yBa1dwvpia 2023-10-05T19:31:14.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2023-10-05T19:31:14.766 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-05T19:31:14.767 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-05T19:31:14.767 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-05T19:31:14.767 INFO:teuthology.orchestra.run.smithi172.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-05T19:31:14.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:31:14.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2023-10-05T19:31:44.966 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:31:44.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:31:44.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:32:19.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-05T19:32:28.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:32:28.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:32:46.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-10-05T19:32:58.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:32:58.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:32:59.173 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-05T19:32:59.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:32:59.429 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-05T19:32:59.429 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-10-05T19:32:59.443 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-05T19:32:59.444 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:32:59.444 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:32:59.444 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-05T19:33:00.445 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T19:33:00.446 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:33:00.587 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-10-05T19:33:00.601 DEBUG:teuthology.parallel:result is None 2023-10-05T19:33:00.602 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T19:33:00.610 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T19:33:00.611 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:33:00.625 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T19:33:00.632 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T19:33:00.640 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T19:33:00.641 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:33:00.708 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T19:33:00.714 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T19:33:00.715 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:33:00.780 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:33:00.791 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:33:00.792 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T19:33:00.800 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T19:33:00.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:33:00.858 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T19:33:00.867 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T19:33:00.867 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:33:00.912 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:33:00.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:33:01.036 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:33:01.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:33:01.157 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:33:01.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:33:01.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-10-05T19:33:01.339 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T19:33:01.456 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T19:33:01.464 INFO:teuthology.task.internal:Starting timer... 2023-10-05T19:33:01.464 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T19:33:01.474 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T19:33:01.483 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-05T19:33:01.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-10-05T19:33:01.526 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: [ OK ] 2023-10-05T19:33:01.528 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T19:33:01.528 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-10-05T19:33:01.585 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-10-05T19:33:01.585 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2023-10-05T19:33:01.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T19:33:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:33:01.650 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T19:33:01.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-10-05T19:33:01.716 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T19:33:01.726 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-05T19:33:01.731 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-05T19:33:01.855 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T19:33:01.870 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T19:33:01.871 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T19:41:02.522 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2023-10-05T19:41:02.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:41:02.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:41:02.593 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-10-05T19:41:02.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-10-05T19:41:02.658 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T19:41:02.669 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T19:41:02.670 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:41:02.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:41:02.728 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-05T19:41:02.742 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-05T19:41:02.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2023-10-05T19:41:02.797 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2023-10-05T19:41:02.811 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-05T19:41:02.824 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-05T19:41:02.883 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-05T19:41:02.885 INFO:teuthology.orchestra.run.smithi172.stdout:210 Number of sources = 4 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:41:02.886 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:41:02.886 INFO:teuthology.run_tasks:Running task install... 2023-10-05T19:41:02.895 DEBUG:teuthology.task.install:project ceph 2023-10-05T19:41:02.895 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T19:41:02.895 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T19:41:02.895 INFO:teuthology.task.install:Using flavor: default 2023-10-05T19:41:02.901 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T19:41:02.901 INFO:teuthology.task.install:extra packages: [] 2023-10-05T19:41:02.901 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'tag': None, 'wait_for_package': False} 2023-10-05T19:41:02.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:41:03.119 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/ 2023-10-05T19:41:03.119 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1278.g487ffcb5 2023-10-05T19:41:03.234 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-05T19:41:03.234 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:41:03.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-05T19:41:03.261 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-05T19:41:03.261 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/487ffcb5ddb60d29c5688d1a457e812b1b697b90/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-05T19:41:03.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-05T19:41:03.401 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-05T19:41:03.464 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2023-10-05T19:41:03.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2023-10-05T19:41:03.858 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:41:04.171 INFO:teuthology.orchestra.run.smithi172.stdout:44 files removed 2023-10-05T19:41:04.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2023-10-05T19:41:04.570 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:41:05.283 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 278 kB/s | 78 kB 00:00 2023-10-05T19:41:05.624 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-10-05T19:41:05.957 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 4.8 kB/s | 1.4 kB 00:00 2023-10-05T19:41:06.080 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 37 kB/s | 3.5 kB 00:00 2023-10-05T19:41:07.581 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-05T19:41:10.926 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-10-05T19:41:11.284 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 119 MB/s | 33 MB 00:00 2023-10-05T19:41:16.391 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 MB/s | 38 MB 00:00 2023-10-05T19:41:25.179 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2023-10-05T19:41:25.182 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.6 M 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.2 M 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 23 M 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 24 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 797 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 506 k 2023-10-05T19:41:25.183 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.6 M 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 45 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 115 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 191 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 370 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 359 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 106 k 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-05T19:41:25.184 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:Install 21 Packages 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 63 M 2023-10-05T19:41:25.185 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:41:33.810 INFO:teuthology.orchestra.run.smithi172.stdout:(1/23): ceph-base-17.2.6-1278.g487ffcb5.el8.x86 639 kB/s | 5.2 MB 00:08 2023-10-05T19:41:34.114 INFO:teuthology.orchestra.run.smithi172.stdout:(2/23): ceph-selinux-17.2.6-1278.g487ffcb5.el8. 80 kB/s | 24 kB 00:00 2023-10-05T19:41:34.415 INFO:teuthology.orchestra.run.smithi172.stdout:(3/23): ceph-radosgw-17.2.6-1278.g487ffcb5.el8. 1.9 MB/s | 17 MB 00:08 2023-10-05T19:41:34.421 INFO:teuthology.orchestra.run.smithi172.stdout:(4/23): libcephfs2-17.2.6-1278.g487ffcb5.el8.x8 2.5 MB/s | 797 kB 00:00 2023-10-05T19:41:34.501 INFO:teuthology.orchestra.run.smithi172.stdout:(5/23): libradosstriper1-17.2.6-1278.g487ffcb5. 5.8 MB/s | 506 kB 00:00 2023-10-05T19:41:34.744 INFO:teuthology.orchestra.run.smithi172.stdout:(6/23): ceph-common-17.2.6-1278.g487ffcb5.el8.x 2.5 MB/s | 23 MB 00:09 2023-10-05T19:41:34.803 INFO:teuthology.orchestra.run.smithi172.stdout:(7/23): python3-ceph-argparse-17.2.6-1278.g487f 148 kB/s | 45 kB 00:00 2023-10-05T19:41:34.804 INFO:teuthology.orchestra.run.smithi172.stdout:(8/23): python3-ceph-common-17.2.6-1278.g487ffc 1.9 MB/s | 115 kB 00:00 2023-10-05T19:41:34.834 INFO:teuthology.orchestra.run.smithi172.stdout:(9/23): python3-cephfs-17.2.6-1278.g487ffcb5.el 6.1 MB/s | 191 kB 00:00 2023-10-05T19:41:34.870 INFO:teuthology.orchestra.run.smithi172.stdout:(10/23): python3-rados-17.2.6-1278.g487ffcb5.el 5.5 MB/s | 370 kB 00:00 2023-10-05T19:41:34.881 INFO:teuthology.orchestra.run.smithi172.stdout:(11/23): python3-rbd-17.2.6-1278.g487ffcb5.el8. 7.5 MB/s | 359 kB 00:00 2023-10-05T19:41:34.901 INFO:teuthology.orchestra.run.smithi172.stdout:(12/23): python3-rgw-17.2.6-1278.g487ffcb5.el8. 3.5 MB/s | 106 kB 00:00 2023-10-05T19:41:34.986 INFO:teuthology.orchestra.run.smithi172.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 690 kB/s | 59 kB 00:00 2023-10-05T19:41:35.014 INFO:teuthology.orchestra.run.smithi172.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 MB/s | 306 kB 00:00 2023-10-05T19:41:35.031 INFO:teuthology.orchestra.run.smithi172.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.6 MB/s | 75 kB 00:00 2023-10-05T19:41:35.076 INFO:teuthology.orchestra.run.smithi172.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 997 kB/s | 44 kB 00:00 2023-10-05T19:41:35.106 INFO:teuthology.orchestra.run.smithi172.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.0 MB/s | 259 kB 00:00 2023-10-05T19:41:35.133 INFO:teuthology.orchestra.run.smithi172.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 12 MB/s | 353 kB 00:00 2023-10-05T19:41:35.152 INFO:teuthology.orchestra.run.smithi172.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2023-10-05T19:41:35.157 INFO:teuthology.orchestra.run.smithi172.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.9 MB/s | 47 kB 00:00 2023-10-05T19:41:35.274 INFO:teuthology.orchestra.run.smithi172.stdout:(21/23): librgw2-17.2.6-1278.g487ffcb5.el8.x86_ 6.5 MB/s | 5.6 MB 00:00 2023-10-05T19:41:35.360 INFO:teuthology.orchestra.run.smithi172.stdout:(22/23): librados2-17.2.6-1278.g487ffcb5.el8.x8 17 MB/s | 3.5 MB 00:00 2023-10-05T19:41:35.444 INFO:teuthology.orchestra.run.smithi172.stdout:(23/23): librbd1-17.2.6-1278.g487ffcb5.el8.x86_ 13 MB/s | 3.6 MB 00:00 2023-10-05T19:41:35.447 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:41:35.447 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.2 MB/s | 63 MB 00:10 2023-10-05T19:41:35.451 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:41:35.494 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:41:35.494 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:41:36.198 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:41:36.227 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:41:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:41:37.680 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-05T19:41:37.921 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-05T19:41:38.145 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:41:38.240 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:41:38.485 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T19:41:38.754 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-10-05T19:41:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:41:39.370 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:41:39.579 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T19:41:39.647 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T19:41:39.819 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-10-05T19:41:40.037 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-05T19:41:40.241 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-05T19:41:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-10-05T19:41:41.095 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:41:41.152 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:41:41.289 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 11/25 2023-10-05T19:41:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T19:41:41.763 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T19:41:41.994 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 14/25 2023-10-05T19:41:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T19:41:42.240 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T19:41:42.432 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-05T19:41:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-05T19:41:42.931 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-05T19:41:43.117 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 19/25 2023-10-05T19:41:48.163 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:41:48.428 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:41:49.009 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:41:49.328 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T19:41:49.416 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T19:41:49.569 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:42:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:42:07.523 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2023-10-05T19:42:07.523 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2023-10-05T19:42:07.523 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-05T19:42:07.523 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-10-05T19:42:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T19:42:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T19:42:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T19:42:09.550 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T19:42:09.551 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T19:42:10.350 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 6/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 7/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 8/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 9/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 11/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-05T19:42:10.351 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-05T19:42:10.352 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/25 2023-10-05T19:42:11.770 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:42:11.770 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:55 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:42:13.670 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:42:13.781 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2023-10-05T19:42:13.781 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.781 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-05T19:42:13.782 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:13.783 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:42:13.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2023-10-05T19:42:14.289 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:42:15.147 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:59 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:42:16.919 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:42:16.920 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 52 M 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 174 k 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:16.921 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 53 M 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 233 M 2023-10-05T19:42:16.922 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:42:16.991 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 70 kB 00:00 2023-10-05T19:42:17.043 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.0 MB/s | 161 kB 00:00 2023-10-05T19:42:17.077 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2023-10-05T19:42:17.100 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): libcephsqlite-17.2.6-1278.g487ffcb5.el8. 984 kB/s | 174 kB 00:00 2023-10-05T19:42:17.108 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): socat-1.7.3.3-2.el8.x86_64.rpm 9.7 MB/s | 302 kB 00:00 2023-10-05T19:42:19.735 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-test-17.2.6-1278.g487ffcb5.el8.x86_ 18 MB/s | 52 MB 00:02 2023-10-05T19:42:19.738 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:42:19.738 INFO:teuthology.orchestra.run.smithi172.stdout:Total 19 MB/s | 53 MB 00:02 2023-10-05T19:42:19.742 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:42:19.764 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:42:19.764 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:42:20.100 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:42:20.104 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:42:20.602 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:42:20.866 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-10-05T19:42:21.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-05T19:42:21.124 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-05T19:42:21.292 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-10-05T19:42:21.478 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-05T19:42:21.679 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T19:42:26.706 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T19:42:26.977 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T19:42:27.245 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T19:42:27.245 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/6 2023-10-05T19:42:27.246 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/6 2023-10-05T19:42:27.246 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-05T19:42:27.246 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-10-05T19:42:27.246 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-05T19:42:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-10-05T19:42:28.364 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:12 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:42:30.220 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:42:30.318 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.5-12.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-05T19:42:30.319 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:30.320 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:42:30.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2023-10-05T19:42:30.785 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:42:31.649 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:15 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:42:33.444 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:42:33.446 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 6.4 k 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 2.3 M 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 1.6 M 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 238 k 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.9 M 2023-10-05T19:42:33.447 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-05T19:42:33.448 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-05T19:42:33.449 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:Install 28 Packages 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 27 M 2023-10-05T19:42:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 106 M 2023-10-05T19:42:33.451 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:42:33.900 INFO:teuthology.orchestra.run.smithi172.stdout:(1/28): ceph-17.2.6-1278.g487ffcb5.el8.x86_64.r 70 kB/s | 6.4 kB 00:00 2023-10-05T19:42:43.798 INFO:teuthology.orchestra.run.smithi172.stdout:(2/28): ceph-mds-17.2.6-1278.g487ffcb5.el8.x86_ 236 kB/s | 2.3 MB 00:09 2023-10-05T19:42:44.238 INFO:teuthology.orchestra.run.smithi172.stdout:(3/28): ceph-mgr-17.2.6-1278.g487ffcb5.el8.x86_ 154 kB/s | 1.6 MB 00:10 2023-10-05T19:42:44.570 INFO:teuthology.orchestra.run.smithi172.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1278.g487f 719 kB/s | 238 kB 00:00 2023-10-05T19:42:44.993 INFO:teuthology.orchestra.run.smithi172.stdout:(5/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 103 kB/s | 44 kB 00:00 2023-10-05T19:42:45.243 INFO:teuthology.orchestra.run.smithi172.stdout:(6/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 744 kB/s | 185 kB 00:00 2023-10-05T19:42:45.329 INFO:teuthology.orchestra.run.smithi172.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2023-10-05T19:42:45.497 INFO:teuthology.orchestra.run.smithi172.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.2 MB/s | 384 kB 00:00 2023-10-05T19:42:45.581 INFO:teuthology.orchestra.run.smithi172.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-10-05T19:42:45.665 INFO:teuthology.orchestra.run.smithi172.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2023-10-05T19:42:45.685 INFO:teuthology.orchestra.run.smithi172.stdout:(11/28): ceph-mon-17.2.6-1278.g487ffcb5.el8.x86 342 kB/s | 3.9 MB 00:11 2023-10-05T19:42:45.749 INFO:teuthology.orchestra.run.smithi172.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 581 kB/s | 49 kB 00:00 2023-10-05T19:42:45.836 INFO:teuthology.orchestra.run.smithi172.stdout:(13/28): python3-pecan-1.3.2-9.el8.noarch.rpm 3.2 MB/s | 283 kB 00:00 2023-10-05T19:42:45.920 INFO:teuthology.orchestra.run.smithi172.stdout:(14/28): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-10-05T19:42:46.004 INFO:teuthology.orchestra.run.smithi172.stdout:(15/28): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2023-10-05T19:42:46.156 INFO:teuthology.orchestra.run.smithi172.stdout:(16/28): ceph-osd-17.2.6-1278.g487ffcb5.el8.x86 7.1 MB/s | 17 MB 00:02 2023-10-05T19:42:46.158 INFO:teuthology.orchestra.run.smithi172.stdout:(17/28): python3-more-itertools-7.2.0-3.el8.noa 124 kB/s | 59 kB 00:00 2023-10-05T19:42:46.158 INFO:teuthology.orchestra.run.smithi172.stdout:(18/28): python3-singledispatch-3.4.0.3-18.el8. 155 kB/s | 24 kB 00:00 2023-10-05T19:42:46.261 INFO:teuthology.orchestra.run.smithi172.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 267 kB/s | 27 kB 00:00 2023-10-05T19:42:46.352 INFO:teuthology.orchestra.run.smithi172.stdout:(20/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2023-10-05T19:42:46.429 INFO:teuthology.orchestra.run.smithi172.stdout:(21/28): python3-waitress-1.4.3-1.el8.noarch.rp 943 kB/s | 255 kB 00:00 2023-10-05T19:42:46.439 INFO:teuthology.orchestra.run.smithi172.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 991 kB/s | 85 kB 00:00 2023-10-05T19:42:46.489 INFO:teuthology.orchestra.run.smithi172.stdout:(23/28): python3-werkzeug-0.12.2-4.el8.noarch.r 8.9 MB/s | 457 kB 00:00 2023-10-05T19:42:46.516 INFO:teuthology.orchestra.run.smithi172.stdout:(24/28): python3-markupsafe-0.23-19.el8.x86_64. 1.4 MB/s | 39 kB 00:00 2023-10-05T19:42:46.522 INFO:teuthology.orchestra.run.smithi172.stdout:(25/28): python3-zc-lockfile-2.0-2.el8.noarch.r 247 kB/s | 23 kB 00:00 2023-10-05T19:42:46.536 INFO:teuthology.orchestra.run.smithi172.stdout:(26/28): python3-tempora-1.14.1-5.el8.noarch.rp 77 kB/s | 29 kB 00:00 2023-10-05T19:42:46.543 INFO:teuthology.orchestra.run.smithi172.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 5.8 MB/s | 157 kB 00:00 2023-10-05T19:42:46.563 INFO:teuthology.orchestra.run.smithi172.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-05T19:42:46.569 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:42:46.570 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.1 MB/s | 27 MB 00:13 2023-10-05T19:42:46.575 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:42:46.614 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:42:46.614 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:42:47.008 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:42:47.013 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:42:48.051 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:42:48.284 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-10-05T19:42:48.490 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-10-05T19:42:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-10-05T19:42:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-10-05T19:42:49.363 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-10-05T19:42:49.558 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-10-05T19:42:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-10-05T19:42:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-10-05T19:42:50.231 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-10-05T19:42:50.430 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-10-05T19:42:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-10-05T19:42:50.844 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-10-05T19:42:50.989 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-10-05T19:42:51.205 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-10-05T19:42:51.420 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-10-05T19:42:51.613 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-10-05T19:42:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-10-05T19:42:52.140 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-10-05T19:42:52.385 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-10-05T19:42:52.685 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-10-05T19:42:52.948 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-10-05T19:42:54.236 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-10-05T19:42:54.598 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 23/28 2023-10-05T19:42:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/28 2023-10-05T19:42:56.251 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/28 2023-10-05T19:42:56.450 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 25/28 2023-10-05T19:42:56.853 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 25/28 2023-10-05T19:42:57.048 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 26/28 2023-10-05T19:42:57.268 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 26/28 2023-10-05T19:42:57.465 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 27/28 2023-10-05T19:42:57.483 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 27/28 2023-10-05T19:42:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/28 2023-10-05T19:42:58.006 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 6/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-10-05T19:42:58.007 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-10-05T19:42:58.008 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-10-05T19:42:58.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-10-05T19:42:58.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-10-05T19:42:58.009 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-10-05T19:42:59.512 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-10-05T19:42:59.707 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:43 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:01.381 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:01.480 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:01.480 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-05T19:43:01.481 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2023-10-05T19:43:01.482 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:01.483 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:01.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2023-10-05T19:43:02.011 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:02.867 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:46 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:04.183 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:43:04.656 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:04.657 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:43:04.657 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:04.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2023-10-05T19:43:05.112 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:05.972 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:49 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:07.759 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 80 k 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:43:07.760 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:07.761 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 80 k 2023-10-05T19:43:07.761 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 364 k 2023-10-05T19:43:07.761 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:43:07.984 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-17.2.6-1278.g487ffcb5.el8.noarch.rpm 359 kB/s | 80 kB 00:00 2023-10-05T19:43:07.985 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:43:07.985 INFO:teuthology.orchestra.run.smithi172.stdout:Total 357 kB/s | 80 kB 00:00 2023-10-05T19:43:07.985 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:43:07.993 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:43:07.993 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:43:07.995 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:43:08.000 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:43:08.057 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:43:10.963 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:43:11.127 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:43:11.338 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:43:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:43:12.355 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:56 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:14.229 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:14.326 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:14.326 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:14.326 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:14.326 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:14.326 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:14.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2023-10-05T19:43:14.836 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:15.679 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:59 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:17.475 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 162 k 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:43:17.476 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:17.477 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 162 k 2023-10-05T19:43:17.477 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 490 k 2023-10-05T19:43:17.477 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:43:17.750 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-17.2.6-1278.g487ffc 592 kB/s | 162 kB 00:00 2023-10-05T19:43:17.750 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:43:17.750 INFO:teuthology.orchestra.run.smithi172.stdout:Total 589 kB/s | 162 kB 00:00 2023-10-05T19:43:17.751 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:43:17.763 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:43:17.763 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:43:17.816 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:43:17.821 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:43:17.935 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:43:18.103 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:43:18.541 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:43:19.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:43:19.571 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:03 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:21.418 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:21.516 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:21.516 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:21.516 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:43:21.516 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:21.516 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:21.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2023-10-05T19:43:22.004 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:22.927 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:06 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:24.230 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:43:24.704 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:24.705 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:43:24.705 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:24.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2023-10-05T19:43:25.166 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:26.028 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:10 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:27.884 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 1.7 M 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:43:28.125 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 23 k 2023-10-05T19:43:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 14 k 2023-10-05T19:43:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-05T19:43:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-05T19:43:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-05T19:43:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:28.137 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:43:28.137 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-10-05T19:43:28.137 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-10-05T19:43:28.137 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.0 M 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 8.0 M 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1278.g487 193 kB/s | 23 kB 00:00 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 1.2 MB/s | 34 kB 00:00 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-routes-2.4.1-12.el8.noarch.rpm 6.3 MB/s | 196 kB 00:00 2023-10-05T19:43:28.138 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): ceph-prometheus-alerts-17.2.6-1278.g487f 73 kB/s | 14 kB 00:00 2023-10-05T19:43:28.141 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 974 kB/s | 43 kB 00:00 2023-10-05T19:43:28.326 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1278.g487ffcb5 4.2 MB/s | 1.7 MB 00:00 2023-10-05T19:43:28.328 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:43:28.334 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.5 MB/s | 2.0 MB 00:00 2023-10-05T19:43:28.334 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:43:28.338 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:43:28.338 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:43:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:43:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:43:28.682 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:43:28.884 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-05T19:43:29.127 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-05T19:43:29.304 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-05T19:43:29.398 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 4/6 2023-10-05T19:43:29.746 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 5/6 2023-10-05T19:43:30.002 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 1/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 2/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 3/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-05T19:43:30.261 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-05T19:43:31.254 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-05T19:43:31.255 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:15 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:33.191 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:33.291 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:33.292 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:33.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-05T19:43:33.781 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:34.631 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:18 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:36.473 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================================================== 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================================================== 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 7.4 M 2023-10-05T19:43:36.474 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================================================== 2023-10-05T19:43:36.475 INFO:teuthology.orchestra.run.smithi172.stdout:Install 8 Packages 2023-10-05T19:43:36.476 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:36.476 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 36 M 2023-10-05T19:43:36.476 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 199 M 2023-10-05T19:43:36.476 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:43:36.522 INFO:teuthology.orchestra.run.smithi172.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 4.9 MB/s | 225 kB 00:00 2023-10-05T19:43:36.600 INFO:teuthology.orchestra.run.smithi172.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 48 MB/s | 3.7 MB 00:00 2023-10-05T19:43:36.693 INFO:teuthology.orchestra.run.smithi172.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 51 MB/s | 4.7 MB 00:00 2023-10-05T19:43:36.809 INFO:teuthology.orchestra.run.smithi172.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 44 MB/s | 15 MB 00:00 2023-10-05T19:43:36.843 INFO:teuthology.orchestra.run.smithi172.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.9 MB/s | 169 kB 00:00 2023-10-05T19:43:36.867 INFO:teuthology.orchestra.run.smithi172.stdout:(6/8): openblas-0.3.12-1.el8.x86_64.rpm 26 MB/s | 4.6 MB 00:00 2023-10-05T19:43:36.874 INFO:teuthology.orchestra.run.smithi172.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2023-10-05T19:43:36.980 INFO:teuthology.orchestra.run.smithi172.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-127 15 MB/s | 7.4 MB 00:00 2023-10-05T19:43:36.981 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:43:36.981 INFO:teuthology.orchestra.run.smithi172.stdout:Total 71 MB/s | 36 MB 00:00 2023-10-05T19:43:36.991 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:43:37.010 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:43:37.010 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:43:37.427 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:43:37.431 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:43:38.036 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:43:38.258 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-05T19:43:38.392 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-05T19:43:38.566 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-05T19:43:39.103 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-05T19:43:39.274 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-05T19:43:39.816 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-05T19:43:39.999 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-10-05T19:43:40.728 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-10-05T19:43:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-10-05T19:43:41.274 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-05T19:43:43.351 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-05T19:43:44.591 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-10-05T19:43:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T19:43:45.064 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 1/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-10-05T19:43:45.065 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-10-05T19:43:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-10-05T19:43:46.077 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:30 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:47.933 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:48.034 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-05T19:43:48.224 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-05T19:43:48.225 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-05T19:43:48.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-05T19:43:48.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-05T19:43:48.226 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:48.226 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:48.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2023-10-05T19:43:48.593 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:49.435 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:33 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:51.257 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:43:51.259 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-10-05T19:43:51.259 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:43:51.259 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-10-05T19:43:51.259 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 47 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-05T19:43:51.260 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2023-10-05T19:43:51.261 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:51.262 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.9 M 2023-10-05T19:43:51.262 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 25 M 2023-10-05T19:43:51.262 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:43:51.350 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 579 kB/s | 33 kB 00:00 2023-10-05T19:43:51.356 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 258 kB/s | 16 kB 00:00 2023-10-05T19:43:51.374 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 3.3 MB/s | 81 kB 00:00 2023-10-05T19:43:51.392 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 3.5 MB/s | 64 kB 00:00 2023-10-05T19:43:51.405 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 4.8 MB/s | 61 kB 00:00 2023-10-05T19:43:51.429 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 16 MB/s | 1.2 MB 00:00 2023-10-05T19:43:51.462 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 470 kB/s | 27 kB 00:00 2023-10-05T19:43:51.482 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.4 MB/s | 126 kB 00:00 2023-10-05T19:43:51.497 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.0 MB/s | 110 kB 00:00 2023-10-05T19:43:51.508 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 770 kB/s | 20 kB 00:00 2023-10-05T19:43:51.526 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): ceph-mgr-rook-17.2.6-1278.g487ffcb5.el 203 kB/s | 47 kB 00:00 2023-10-05T19:43:51.532 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 4.4 MB/s | 155 kB 00:00 2023-10-05T19:43:51.538 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 1.4 MB/s | 43 kB 00:00 2023-10-05T19:43:51.541 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:43:51.541 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.9 MB/s | 1.9 MB 00:00 2023-10-05T19:43:51.547 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:43:51.561 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:43:51.561 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:43:51.651 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:43:51.661 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:43:52.154 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:43:52.484 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-05T19:43:52.703 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-05T19:43:52.957 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-05T19:43:53.245 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-10-05T19:43:53.470 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-10-05T19:43:53.679 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-10-05T19:43:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-10-05T19:43:54.154 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-05T19:43:54.362 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-05T19:43:54.595 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-05T19:43:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-05T19:43:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-05T19:43:56.204 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T19:43:56.474 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T19:43:56.474 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 1/13 2023-10-05T19:43:56.474 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-10-05T19:43:56.475 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-05T19:43:56.476 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-05T19:43:56.476 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-10-05T19:43:57.561 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-10-05T19:43:57.561 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:41 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:43:59.443 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:43:59.543 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-05T19:43:59.544 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-05T19:43:59.545 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-05T19:43:59.545 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-05T19:43:59.545 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-05T19:43:59.545 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:43:59.545 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:43:59.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2023-10-05T19:44:00.036 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:00.882 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:44 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:02.747 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 121 k 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-05T19:44:02.748 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2023-10-05T19:44:02.749 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:02.750 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.0 M 2023-10-05T19:44:02.750 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2023-10-05T19:44:02.750 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:44:03.034 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 493 kB/s | 60 kB 00:00 2023-10-05T19:44:03.093 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.6 MB/s | 476 kB 00:00 2023-10-05T19:44:03.132 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2023-10-05T19:44:03.143 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 10 MB/s | 538 kB 00:00 2023-10-05T19:44:03.191 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): ceph-mgr-cephadm-17.2.6-1278.g487ffcb5.e 433 kB/s | 121 kB 00:00 2023-10-05T19:44:03.194 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:44:03.194 INFO:teuthology.orchestra.run.smithi172.stdout:Total 13 MB/s | 6.0 MB 00:00 2023-10-05T19:44:03.197 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:44:03.207 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:44:03.207 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:44:03.304 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:44:03.308 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:44:04.307 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:44:04.653 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-10-05T19:44:04.907 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-10-05T19:44:05.246 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-05T19:44:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-05T19:44:05.686 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T19:44:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T19:44:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/5 2023-10-05T19:44:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-05T19:44:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-05T19:44:05.954 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-10-05T19:44:06.962 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-10-05T19:44:06.962 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:50 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:08.914 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-05T19:44:09.014 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-05T19:44:09.015 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:09.015 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:09.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2023-10-05T19:44:09.516 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:10.361 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:54 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:12.182 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 876 k 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-05T19:44:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 958 k 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 3.1 M 2023-10-05T19:44:12.184 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:44:12.227 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-10-05T19:44:12.562 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-17.2.6-1278.g487ffcb5.el8.x86_ 2.3 MB/s | 876 kB 00:00 2023-10-05T19:44:12.563 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:44:12.564 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.5 MB/s | 958 kB 00:00 2023-10-05T19:44:12.565 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:44:12.586 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:44:12.586 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:44:12.669 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:44:12.673 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:44:12.924 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:44:13.968 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-10-05T19:44:15.002 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T19:44:15.391 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T19:44:15.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/2 2023-10-05T19:44:16.435 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-10-05T19:44:16.436 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:00 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:18.283 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:44:18.380 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:18.890 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:44:18.890 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-10-05T19:44:18.890 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:18.891 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:18.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2023-10-05T19:44:19.273 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:20.121 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:04 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:21.987 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:21.988 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:21.988 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:44:21.988 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:21.988 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 267 k 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:21.989 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 267 k 2023-10-05T19:44:21.990 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.3 M 2023-10-05T19:44:21.990 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:44:22.165 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-volume-17.2.6-1278.g487ffcb5.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-10-05T19:44:22.165 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:44:22.166 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-10-05T19:44:22.166 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:44:22.177 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:44:22.177 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:44:22.203 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:44:22.207 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:44:22.387 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:44:22.667 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:44:23.068 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:44:24.106 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:44:24.106 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:08 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:26.035 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:44:26.132 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:26.133 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:44:26.133 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:44:26.133 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:26.133 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:26.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2023-10-05T19:44:26.623 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:27.491 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:11 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:29.296 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 130 k 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:29.340 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 130 k 2023-10-05T19:44:29.341 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 464 k 2023-10-05T19:44:29.341 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:44:29.451 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-17.2.6-1278.g487ffcb5.el8.x86_64 848 kB/s | 130 kB 00:00 2023-10-05T19:44:29.451 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:44:29.454 INFO:teuthology.orchestra.run.smithi172.stdout:Total 840 kB/s | 130 kB 00:00 2023-10-05T19:44:29.454 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:44:29.459 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:44:29.459 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:44:29.515 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:44:29.519 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:44:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:44:29.848 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:30.116 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:31.224 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:31.224 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:15 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:33.123 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:44:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:33.221 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:44:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:44:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:33.221 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:33.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2023-10-05T19:44:33.711 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:34.557 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:18 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:35.953 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:36.423 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:36.424 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:36.424 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:36.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2023-10-05T19:44:36.894 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:37.754 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:21 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:39.554 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:39.554 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 25 k 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:44:39.555 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:39.556 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 25 k 2023-10-05T19:44:39.556 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 109 k 2023-10-05T19:44:39.556 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:44:39.744 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-17.2.6-1278.g487ffcb5.el8.x86_6 132 kB/s | 25 kB 00:00 2023-10-05T19:44:39.744 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:44:39.745 INFO:teuthology.orchestra.run.smithi172.stdout:Total 131 kB/s | 25 kB 00:00 2023-10-05T19:44:39.745 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:44:39.748 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:44:39.748 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:44:39.750 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:44:39.755 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:44:39.837 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:44:40.012 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:40.253 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:41.229 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:44:41.229 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:25 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:43.071 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:44:43.167 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:43.168 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:44:43.168 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:44:43.168 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:44:43.168 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:43.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2023-10-05T19:44:43.656 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:44.518 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:28 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:45.849 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:46.327 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:46.327 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:46.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2023-10-05T19:44:46.804 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:47.635 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:31 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:48.997 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:49.471 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:49.472 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:49.473 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:49.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2023-10-05T19:44:49.950 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:50.798 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:34 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:52.189 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:52.657 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:52.658 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:52.658 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:52.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2023-10-05T19:44:53.150 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:54.015 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:38 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:55.431 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:55.926 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:55.928 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:55.928 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:56.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2023-10-05T19:44:56.445 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:44:57.826 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:41 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:44:59.214 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:44:59.720 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:44:59.721 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:44:59.721 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:44:59.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2023-10-05T19:45:00.240 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:45:01.102 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:45 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:02.555 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:45:03.047 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:45:03.048 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-10-05T19:45:03.049 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:45:03.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2023-10-05T19:45:03.542 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:45:04.413 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:48 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:06.300 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 90 k 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:45:06.301 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:06.302 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 90 k 2023-10-05T19:45:06.302 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 264 k 2023-10-05T19:45:06.302 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:45:06.563 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 346 kB/s | 90 kB 00:00 2023-10-05T19:45:06.564 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:45:06.564 INFO:teuthology.orchestra.run.smithi172.stdout:Total 344 kB/s | 90 kB 00:00 2023-10-05T19:45:06.564 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:45:06.577 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:45:06.577 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:45:06.636 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:45:06.640 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:45:06.784 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:45:06.934 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:07.210 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:08.235 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:08.235 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:52 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:10.142 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:45:10.238 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:10.238 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:45:10.238 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:45:10.238 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:10.238 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:45:10.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2023-10-05T19:45:10.730 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:45:11.604 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:55 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:13.415 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:45:13.415 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:13.415 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-10-05T19:45:13.415 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:13.415 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:13.416 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.5 M 2023-10-05T19:45:13.417 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 16 M 2023-10-05T19:45:13.417 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:45:13.882 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 7.4 MB/s | 3.5 MB 00:00 2023-10-05T19:45:13.883 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:45:13.884 INFO:teuthology.orchestra.run.smithi172.stdout:Total 7.4 MB/s | 3.5 MB 00:00 2023-10-05T19:45:13.884 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:45:13.897 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:45:13.898 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:45:13.968 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:45:13.972 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:45:14.429 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:45:14.616 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:15.038 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:16.054 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:16.055 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:00 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:18.027 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:45:18.124 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:18.125 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:45:18.125 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:45:18.125 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:18.125 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:45:18.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2023-10-05T19:45:18.623 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:45:19.529 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:03 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:21.354 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:45:21.354 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:21.354 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-10-05T19:45:21.354 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:21.354 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 173 k 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-10-05T19:45:21.355 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:21.356 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 173 k 2023-10-05T19:45:21.356 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 552 k 2023-10-05T19:45:21.356 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:45:21.618 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 658 kB/s | 173 kB 00:00 2023-10-05T19:45:21.619 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:45:21.620 INFO:teuthology.orchestra.run.smithi172.stdout:Total 653 kB/s | 173 kB 00:00 2023-10-05T19:45:21.620 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:45:21.632 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:45:21.633 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:45:21.688 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:45:21.693 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:45:21.815 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:45:22.005 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:22.273 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:23.438 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:45:23.438 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:07 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:45:25.317 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:45:25.417 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:25.417 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:45:25.417 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:45:25.418 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:25.418 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:45:25.532 DEBUG:teuthology.parallel:result is None 2023-10-05T19:45:25.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:45:25.678 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-05T19:45:25.697 INFO:teuthology.orchestra.run.smithi172.stdout:17.2.6-1278.g487ffcb5.el8 2023-10-05T19:45:25.697 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278.g487ffcb5.el8 2023-10-05T19:45:25.698 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278.g487ffcb5 is installed. 2023-10-05T19:45:25.698 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T19:45:25.698 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:25.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T19:45:25.762 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T19:45:25.763 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:25.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T19:45:25.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T19:45:25.890 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T19:45:25.891 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:25.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T19:45:25.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T19:45:26.018 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T19:45:26.018 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:26.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T19:45:26.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T19:45:26.147 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T19:45:26.207 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T19:45:26.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2023-10-05T19:45:26.231 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T19:45:26.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T19:45:26.294 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T19:45:26.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T19:45:26.367 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T19:45:26.367 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T19:45:26.368 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411761', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvnFQtyyPRYWPgri6uc34sDcJemY45OMhSqNvFCy5pir0u3AHWCvD56dzwRnNfAz+Y0TckFOlVUYp/kjtEG/EA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'kv', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/kv/data/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1', 'secret': 'a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c='}, {'path': 'my-key-2', 'secret': 'a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg='}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T19:45:26.368 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T19:45:26.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T19:45:26.479 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:26.479 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T19:45:26.535 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T19:45:26.535 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-10-05T19:45:26.589 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 219380 Links: 1 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-05 19:45:14.802060898 +0000 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-05 19:40:34.717201646 +0000 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-05 19:40:34.717201646 +0000 2023-10-05T19:45:26.590 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-05T19:45:26.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T19:45:26.653 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-05T19:45:26.653 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-05T19:45:26.653 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000232424 s, 2.2 MB/s 2023-10-05T19:45:26.654 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T19:45:26.710 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-10-05T19:45:26.765 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 219030 Links: 1 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-05 19:45:14.802060898 +0000 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-05 19:40:35.209190048 +0000 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-05 19:40:35.209190048 +0000 2023-10-05T19:45:26.766 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-05T19:45:26.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T19:45:26.829 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-05T19:45:26.829 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-05T19:45:26.829 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000218124 s, 2.3 MB/s 2023-10-05T19:45:26.830 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T19:45:26.885 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 219481 Links: 1 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-05 19:45:14.802060898 +0000 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-05 19:40:35.668179228 +0000 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-05 19:40:35.668179228 +0000 2023-10-05T19:45:26.940 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-05T19:45:26.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T19:45:27.003 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-05T19:45:27.003 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-05T19:45:27.003 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000227412 s, 2.3 MB/s 2023-10-05T19:45:27.004 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T19:45:27.059 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-10-05T19:45:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T19:45:27.114 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:45:27.114 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 215745 Links: 1 2023-10-05T19:45:27.114 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:45:27.114 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:45:27.115 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-10-05 19:45:14.802060898 +0000 2023-10-05T19:45:27.115 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-10-05 19:40:36.143168030 +0000 2023-10-05T19:45:27.115 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-10-05 19:40:36.143168030 +0000 2023-10-05T19:45:27.115 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-10-05T19:45:27.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T19:45:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-10-05T19:45:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-10-05T19:45:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000201296 s, 2.5 MB/s 2023-10-05T19:45:27.178 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T19:45:27.233 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:45:27.233 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:45:27.233 INFO:tasks.ceph:Generating config... 2023-10-05T19:45:27.234 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/kv/data 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] rgw crypt vault secret engine = kv 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T19:45:27.235 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T19:45:27.235 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T19:45:27.235 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T19:45:27.235 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T19:45:27.235 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T19:45:27.236 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T19:45:27.236 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T19:45:27.236 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T19:45:27.236 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-05T19:45:27.236 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T19:45:27.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T19:45:27.333 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T19:45:27.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T19:45:27.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:45:27.439 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.172')] 2023-10-05T19:45:27.439 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/kv/data', 'rgw crypt vault secret engine': 'kv', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T19:45:27.440 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:27.440 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:45:27.493 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:27.569 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:45:27.569 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:45:27.569 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 5e13a587-bd3a-4c44-ab2a-e881fd60ca16 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = octopus 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 5e13a587-bd3a-4c44-ab2a-e881fd60ca16 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2023-10-05T19:45:27.568810+0000 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:created 2023-10-05T19:45:27.568810+0000 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 15 (octopus) 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2023-10-05T19:45:27.570 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T19:45:27.571 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:45:27.626 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5e13a587-bd3a-4c44-ab2a-e881fd60ca16... 2023-10-05T19:45:27.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T19:45:27.709 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-10-05T19:45:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2023-10-05T19:45:27.709 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2023-10-05T19:45:27.710 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2023-10-05T19:45:27.711 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.712 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 5e13a587-bd3a-4c44-ab2a-e881fd60ca16 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2023-10-05T19:45:27.713 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.714 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-05T19:45:27.715 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2023-10-05T19:45:27.716 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T19:45:27.717 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2023-10-05T19:45:27.718 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 20 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt require ssl = False 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt s3 kms backend = vault 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt vault auth = token 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt vault prefix = /v1/kv/data 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: rgw crypt vault secret engine = kv 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: setgroup = ceph 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout: setuser = ceph 2023-10-05T19:45:27.719 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2023-10-05T19:45:27.721 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T19:45:27.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T19:45:27.811 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T19:45:27.812 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:27.812 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T19:45:27.826 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:27.826 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T19:45:27.881 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:45:27.881 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:27.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T19:45:27.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:45:27.954 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:27.954 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:28.009 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T19:45:28.009 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T19:45:28.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:45:28.096 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:45:28.098 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T19:45:28.099 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T19:45:28.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T19:45:28.136 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T19:45:28.148 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T19:45:28.149 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:45:28.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T19:45:28.213 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:45:28.213 INFO:tasks.ceph:role: osd.0 2023-10-05T19:45:28.213 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:45:28.214 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T19:45:28.285 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:45:28.286 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:45:28.722 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-05T19:45:28.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-05T19:45:28.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T19:45:28.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T19:45:28.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T19:45:28.968 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:45:28.969 INFO:tasks.ceph:role: osd.1 2023-10-05T19:45:28.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:45:28.969 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T19:45:29.046 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:45:29.047 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:45:29.465 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-05T19:45:29.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-05T19:45:29.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-05T19:45:29.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T19:45:29.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T19:45:29.679 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:45:29.679 INFO:tasks.ceph:role: osd.2 2023-10-05T19:45:29.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:45:29.680 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T19:45:29.751 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:45:29.751 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:45:29.751 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:45:29.752 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:45:30.192 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2023-10-05T19:45:30.193 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2023-10-05T19:45:30.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-05T19:45:30.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T19:45:30.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.414+0000 7f0b3770f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T19:45:30.419 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.416+0000 7f0b3770f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T19:45:30.419 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.416+0000 7f0b3770f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:45:30.419 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.417+0000 7f0b3770f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:45:30.419 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.417+0000 7f0b3770f540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:45:30.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.421+0000 7f0b3770f540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T19:45:30.953 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:30.951+0000 7f0b3770f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:31.712 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:31.710+0000 7f0b3770f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:32.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T19:45:32.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:32.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.605+0000 7f337b133540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T19:45:32.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.605+0000 7f337b133540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T19:45:32.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.605+0000 7f337b133540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:45:32.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.605+0000 7f337b133540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:45:32.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.605+0000 7f337b133540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:45:32.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:32.606+0000 7f337b133540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T19:45:33.130 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:33.128+0000 7f337b133540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:33.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:33.893+0000 7f337b133540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:34.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T19:45:34.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:34.839 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.837+0000 7fe866c65540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T19:45:34.839 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.837+0000 7fe866c65540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T19:45:34.839 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.837+0000 7fe866c65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:45:34.840 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.837+0000 7fe866c65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:45:34.840 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.837+0000 7fe866c65540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:45:34.840 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:34.838+0000 7fe866c65540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T19:45:35.361 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:35.359+0000 7fe866c65540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:36.128 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:45:36.126+0000 7fe866c65540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:45:36.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T19:45:36.983 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T19:45:36.983 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:36.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T19:45:37.047 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:37.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T19:45:37.112 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:37.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T19:45:37.176 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:37.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T19:45:37.239 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:37.240 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T19:45:37.294 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T19:45:37.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T19:45:37.356 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2023-10-05T19:45:37.356 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBYEh9lsvO1BRAAcMN4Hou3p4/F0huQFU6cpw== 2023-10-05T19:45:37.356 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBaEh9luinCGBAAcR41caynzeMNofl/6cN7pg== 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBcEh9lFGgoJBAASpTq34giOZv+809hTkBGCQ== 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBeEh9lR5T6MRAAyZzKCHPvtQAuDYmAj/FlHg== 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2023-10-05T19:45:37.357 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBYEh9lnP0bCBAA/d+VUaRkdx5LIxev5ZNGCQ== 2023-10-05T19:45:37.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T19:45:37.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:45:37.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:45:37.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:45:37.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T19:45:37.634 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T19:45:37.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T19:45:37.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T19:45:37.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T19:45:37.774 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:45:37.828 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T19:45:37.828 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T19:45:37.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T19:45:37.870 INFO:tasks.ceph.mon.a:Started 2023-10-05T19:45:37.870 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T19:45:37.871 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T19:45:37.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T19:45:37.873 INFO:tasks.ceph.mgr.x:Started 2023-10-05T19:45:37.873 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T19:45:37.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T19:45:38.098 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2023-10-05T19:45:38.109 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T19:45:38.109 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:38.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T19:45:38.134 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:38.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T19:45:38.198 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:45:38.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T19:45:38.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 4601c289-3389-47b7-b869-91f746bd4764 0 2023-10-05T19:45:38.512 INFO:teuthology.orchestra.run.smithi172.stdout:0 2023-10-05T19:45:38.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 624131d9-ccda-440a-b258-3df6734b0890 1 2023-10-05T19:45:38.732 INFO:teuthology.orchestra.run.smithi172.stdout:1 2023-10-05T19:45:38.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 26fa6f73-54a9-4005-b2ee-02f42fe65f8c 2 2023-10-05T19:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout:2 2023-10-05T19:45:38.964 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T19:45:38.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T19:45:38.966 INFO:tasks.ceph.osd.0:Started 2023-10-05T19:45:38.966 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T19:45:38.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T19:45:38.968 INFO:tasks.ceph.osd.1:Started 2023-10-05T19:45:38.968 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T19:45:38.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T19:45:38.970 INFO:tasks.ceph.osd.2:Started 2023-10-05T19:45:38.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:45:39.172 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:39.172 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:45:38.948574+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:45:39.185 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T19:45:39.185 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T19:45:39.379 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:39.377+0000 7fcede51c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T19:45:39.487 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:45:39.513 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:39.511+0000 7fcede51c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T19:45:39.568 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f575afce540 -1 Falling back to public interface 2023-10-05T19:45:39.568 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f57553f6700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-10-05T19:45:39.568 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f814d91f540 -1 Falling back to public interface 2023-10-05T19:45:39.568 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f8147d47700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-10-05T19:45:39.569 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f57553f6700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-10-05T19:45:39.569 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:39.566+0000 7f57553f6700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6806/0: (98) Address already in use 2023-10-05T19:45:39.581 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-05T19:45:39.579+0000 7f178c809540 -1 Falling back to public interface 2023-10-05T19:45:39.593 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:39.591+0000 7fcede51c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T19:45:39.676 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-05T19:45:39.676 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:45:38.948574+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:45:39.687 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-05T19:45:39.785 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:39.783+0000 7fcede51c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:45:39.959 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:39.957+0000 7fcede51c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T19:45:40.036 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.034+0000 7fcede51c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T19:45:40.174 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.171+0000 7fcede51c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T19:45:40.269 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.266+0000 7fcede51c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T19:45:40.806 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.804+0000 7fcede51c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T19:45:40.883 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.881+0000 7fcede51c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T19:45:40.962 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:40.960+0000 7fcede51c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T19:45:41.544 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:41.542+0000 7fcede51c180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-05T19:45:41.681 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:41.679+0000 7fcede51c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:45:41.769 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:41.766+0000 7fcede51c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T19:45:42.020 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:42.017+0000 7fcede51c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-05T19:45:42.101 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:42.099+0000 7fcede51c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T19:45:42.287 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:42.285+0000 7f575afce540 -1 osd.0 0 log_to_monitors true 2023-10-05T19:45:42.304 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-05T19:45:42.302+0000 7f814d91f540 -1 osd.1 0 log_to_monitors true 2023-10-05T19:45:42.322 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-05T19:45:42.320+0000 7f178c809540 -1 osd.2 0 log_to_monitors true 2023-10-05T19:45:42.934 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:42.932+0000 7fcede51c180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-05T19:45:43.008 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.006+0000 7fcede51c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T19:45:43.083 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.081+0000 7fcede51c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T19:45:43.215 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.213+0000 7fcede51c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T19:45:43.299 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.297+0000 7fcede51c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T19:45:43.467 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.465+0000 7fcede51c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T19:45:43.551 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.548+0000 7fcede51c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T19:45:43.637 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.635+0000 7fcede51c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T19:45:43.725 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:43.723+0000 7fcede51c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T19:45:44.588 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:44.586+0000 7f5751bef700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T19:45:44.590 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-05T19:45:44.588+0000 7f8144540700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T19:45:44.590 INFO:tasks.ceph.osd.0.smithi172.stderr:2023-10-05T19:45:44.588+0000 7f574a9e6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:45:44.593 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-05T19:45:44.591+0000 7f178342a700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T19:45:44.594 INFO:tasks.ceph.osd.1.smithi172.stderr:2023-10-05T19:45:44.593+0000 7f813d337700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:45:44.596 INFO:tasks.ceph.osd.2.smithi172.stderr:2023-10-05T19:45:44.594+0000 7f177c221700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:45:44.877 INFO:tasks.ceph.mgr.x.smithi172.stderr:2023-10-05T19:45:44.874+0000 7fce9791f700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T19:45:45.989 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:45:46.296 INFO:teuthology.misc.health.smithi172.stdout: 2023-10-05T19:45:46.296 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":8,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:45:45.593853+0000","last_up_change":"2023-10-05T19:45:45.593853+0000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":86111},{"type":"v1","addr":"172.21.15.172:6804","nonce":86111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":86111},{"type":"v1","addr":"172.21.15.172:6809","nonce":86111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":86111},{"type":"v1","addr":"172.21.15.172:6815","nonce":86111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":86111},{"type":"v1","addr":"172.21.15.172:6813","nonce":86111}]},"public_addr":"172.21.15.172:6804/86111","cluster_addr":"172.21.15.172:6809/86111","heartbeat_back_addr":"172.21.15.172:6815/86111","heartbeat_front_addr":"172.21.15.172:6813/86111","state":["exists","up"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":86112},{"type":"v1","addr":"172.21.15.172:6800","nonce":86112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":86112},{"type":"v1","addr":"172.21.15.172:6805","nonce":86112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":86112},{"type":"v1","addr":"172.21.15.172:6811","nonce":86112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":86112},{"type":"v1","addr":"172.21.15.172:6808","nonce":86112}]},"public_addr":"172.21.15.172:6800/86112","cluster_addr":"172.21.15.172:6805/86112","heartbeat_back_addr":"172.21.15.172:6811/86112","heartbeat_front_addr":"172.21.15.172:6808/86112","state":["exists","up"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":86141},{"type":"v1","addr":"172.21.15.172:6817","nonce":86141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":86141},{"type":"v1","addr":"172.21.15.172:6819","nonce":86141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":86141},{"type":"v1","addr":"172.21.15.172:6823","nonce":86141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":86141},{"type":"v1","addr":"172.21.15.172:6821","nonce":86141}]},"public_addr":"172.21.15.172:6817/86141","cluster_addr":"172.21.15.172:6819/86141","heartbeat_back_addr":"172.21.15.172:6823/86141","heartbeat_front_addr":"172.21.15.172:6821/86141","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:45:46.308 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-05T19:45:46.309 INFO:tasks.ceph:Creating RBD pool 2023-10-05T19:45:46.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T19:45:46.885 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2023-10-05T19:45:46.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T19:45:47.885 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T19:45:47.898 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T19:45:47.898 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T19:45:47.899 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T19:45:47.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T19:45:47.899 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T19:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":85773},{"type":"v1","addr":"172.21.15.172:6825","nonce":85773}]},"active_addr":"172.21.15.172:6825/85773","active_change":"2023-10-05T19:45:43.737129+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3734453333}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1824365021}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2683998821}]}]} 2023-10-05T19:45:48.372 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T19:45:48.372 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T19:45:48.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:45:48.681 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:48.681 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":10,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:45:47.877293+0000","last_up_change":"2023-10-05T19:45:45.593853+0000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:45:45.913708+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:45:46.630393+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":86111},{"type":"v1","addr":"172.21.15.172:6804","nonce":86111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":86111},{"type":"v1","addr":"172.21.15.172:6809","nonce":86111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":86111},{"type":"v1","addr":"172.21.15.172:6815","nonce":86111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":86111},{"type":"v1","addr":"172.21.15.172:6813","nonce":86111}]},"public_addr":"172.21.15.172:6804/86111","cluster_addr":"172.21.15.172:6809/86111","heartbeat_back_addr":"172.21.15.172:6815/86111","heartbeat_front_addr":"172.21.15.172:6813/86111","state":["exists","up"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":86112},{"type":"v1","addr":"172.21.15.172:6800","nonce":86112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":86112},{"type":"v1","addr":"172.21.15.172:6805","nonce":86112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":86112},{"type":"v1","addr":"172.21.15.172:6811","nonce":86112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":86112},{"type":"v1","addr":"172.21.15.172:6808","nonce":86112}]},"public_addr":"172.21.15.172:6800/86112","cluster_addr":"172.21.15.172:6805/86112","heartbeat_back_addr":"172.21.15.172:6811/86112","heartbeat_front_addr":"172.21.15.172:6808/86112","state":["exists","up"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":86141},{"type":"v1","addr":"172.21.15.172:6817","nonce":86141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":86141},{"type":"v1","addr":"172.21.15.172:6819","nonce":86141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":86141},{"type":"v1","addr":"172.21.15.172:6823","nonce":86141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":86141},{"type":"v1","addr":"172.21.15.172:6821","nonce":86141}]},"public_addr":"172.21.15.172:6817/86141","cluster_addr":"172.21.15.172:6819/86141","heartbeat_back_addr":"172.21.15.172:6823/86141","heartbeat_front_addr":"172.21.15.172:6821/86141","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.249010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.331326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.316894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:45:48.695 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T19:45:48.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:45:49.003 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:49.003 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":11,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:45:48.878983+0000","last_up_change":"2023-10-05T19:45:45.593853+0000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:45:45.913708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:45:46.630393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":86111},{"type":"v1","addr":"172.21.15.172:6804","nonce":86111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":86111},{"type":"v1","addr":"172.21.15.172:6809","nonce":86111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":86111},{"type":"v1","addr":"172.21.15.172:6815","nonce":86111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":86111},{"type":"v1","addr":"172.21.15.172:6813","nonce":86111}]},"public_addr":"172.21.15.172:6804/86111","cluster_addr":"172.21.15.172:6809/86111","heartbeat_back_addr":"172.21.15.172:6815/86111","heartbeat_front_addr":"172.21.15.172:6813/86111","state":["exists","up"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":86112},{"type":"v1","addr":"172.21.15.172:6800","nonce":86112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":86112},{"type":"v1","addr":"172.21.15.172:6805","nonce":86112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":86112},{"type":"v1","addr":"172.21.15.172:6811","nonce":86112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":86112},{"type":"v1","addr":"172.21.15.172:6808","nonce":86112}]},"public_addr":"172.21.15.172:6800/86112","cluster_addr":"172.21.15.172:6805/86112","heartbeat_back_addr":"172.21.15.172:6811/86112","heartbeat_front_addr":"172.21.15.172:6808/86112","state":["exists","up"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":86141},{"type":"v1","addr":"172.21.15.172:6817","nonce":86141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":86141},{"type":"v1","addr":"172.21.15.172:6819","nonce":86141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":86141},{"type":"v1","addr":"172.21.15.172:6823","nonce":86141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":86141},{"type":"v1","addr":"172.21.15.172:6821","nonce":86141}]},"public_addr":"172.21.15.172:6817/86141","cluster_addr":"172.21.15.172:6819/86141","heartbeat_back_addr":"172.21.15.172:6823/86141","heartbeat_front_addr":"172.21.15.172:6821/86141","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.249010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.331326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.316894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:45:49.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T19:45:49.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T19:45:49.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T19:45:49.181 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-05T19:45:49.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:45:49.187 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-05T19:45:49.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:45:49.197 INFO:teuthology.orchestra.run.smithi172.stdout:34359738371 2023-10-05T19:45:49.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:45:49.511 INFO:teuthology.orchestra.run.smithi172.stdout:34359738369 2023-10-05T19:45:49.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2023-10-05T19:45:49.561 INFO:teuthology.orchestra.run.smithi172.stdout:34359738369 2023-10-05T19:45:49.569 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-05T19:45:49.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-10-05T19:45:49.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.1 2023-10-05T19:45:50.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:45:50.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:45:50.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:45:50.873 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-05T19:45:50.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.2 2023-10-05T19:45:50.888 DEBUG:teuthology.parallel:result is None 2023-10-05T19:45:50.918 INFO:teuthology.orchestra.run.smithi172.stdout:34359738370 2023-10-05T19:45:50.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2023-10-05T19:45:50.932 DEBUG:teuthology.parallel:result is None 2023-10-05T19:45:50.947 INFO:teuthology.orchestra.run.smithi172.stdout:34359738371 2023-10-05T19:45:50.961 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.1 2023-10-05T19:45:50.961 DEBUG:teuthology.parallel:result is None 2023-10-05T19:45:50.962 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T19:45:50.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:45:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:52.645 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-05T19:45:52.646 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-05T19:45:49.873774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893860,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313678940,"statfs":{"total":322122547200,"available":321207234560,"internally_reserved":0,"allocated":1458176,"data_stored":1248104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.889998+0000","last_change":"2023-10-05T19:45:47.888567+0000","last_active":"2023-10-05T19:45:47.889998+0000","last_peered":"2023-10-05T19:45:47.889998+0000","last_clean":"2023-10-05T19:45:47.889998+0000","last_became_active":"2023-10-05T19:45:47.888343+0000","last_became_peered":"2023-10-05T19:45:47.888343+0000","last_unstale":"2023-10-05T19:45:47.889998+0000","last_undegraded":"2023-10-05T19:45:47.889998+0000","last_fullsized":"2023-10-05T19:45:47.889998+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:49.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888786+0000","last_change":"2023-10-05T19:45:47.887584+0000","last_active":"2023-10-05T19:45:47.888786+0000","last_peered":"2023-10-05T19:45:47.888786+0000","last_clean":"2023-10-05T19:45:47.888786+0000","last_became_active":"2023-10-05T19:45:47.887352+0000","last_became_peered":"2023-10-05T19:45:47.887352+0000","last_unstale":"2023-10-05T19:45:47.888786+0000","last_undegraded":"2023-10-05T19:45:47.888786+0000","last_fullsized":"2023-10-05T19:45:47.888786+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:09:31.632762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888783+0000","last_change":"2023-10-05T19:45:47.887688+0000","last_active":"2023-10-05T19:45:47.888783+0000","last_peered":"2023-10-05T19:45:47.888783+0000","last_clean":"2023-10-05T19:45:47.888783+0000","last_became_active":"2023-10-05T19:45:47.887430+0000","last_became_peered":"2023-10-05T19:45:47.887430+0000","last_unstale":"2023-10-05T19:45:47.888783+0000","last_undegraded":"2023-10-05T19:45:47.888783+0000","last_fullsized":"2023-10-05T19:45:47.888783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:44:09.989557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.892882+0000","last_change":"2023-10-05T19:45:47.889252+0000","last_active":"2023-10-05T19:45:47.892882+0000","last_peered":"2023-10-05T19:45:47.892882+0000","last_clean":"2023-10-05T19:45:47.892882+0000","last_became_active":"2023-10-05T19:45:47.889058+0000","last_became_peered":"2023-10-05T19:45:47.889058+0000","last_unstale":"2023-10-05T19:45:47.892882+0000","last_undegraded":"2023-10-05T19:45:47.892882+0000","last_fullsized":"2023-10-05T19:45:47.892882+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:03:39.173092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:48.367886+0000","last_change":"2023-10-05T19:45:47.888051+0000","last_active":"2023-10-05T19:45:48.367886+0000","last_peered":"2023-10-05T19:45:48.367886+0000","last_clean":"2023-10-05T19:45:48.367886+0000","last_became_active":"2023-10-05T19:45:47.887338+0000","last_became_peered":"2023-10-05T19:45:47.887338+0000","last_unstale":"2023-10-05T19:45:48.367886+0000","last_undegraded":"2023-10-05T19:45:48.367886+0000","last_fullsized":"2023-10-05T19:45:48.367886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:18:45.531480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.889985+0000","last_change":"2023-10-05T19:45:47.888336+0000","last_active":"2023-10-05T19:45:47.889985+0000","last_peered":"2023-10-05T19:45:47.889985+0000","last_clean":"2023-10-05T19:45:47.889985+0000","last_became_active":"2023-10-05T19:45:47.887988+0000","last_became_peered":"2023-10-05T19:45:47.887988+0000","last_unstale":"2023-10-05T19:45:47.889985+0000","last_undegraded":"2023-10-05T19:45:47.889985+0000","last_fullsized":"2023-10-05T19:45:47.889985+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:50:58.570285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.892875+0000","last_change":"2023-10-05T19:45:47.888950+0000","last_active":"2023-10-05T19:45:47.892875+0000","last_peered":"2023-10-05T19:45:47.892875+0000","last_clean":"2023-10-05T19:45:47.892875+0000","last_became_active":"2023-10-05T19:45:47.888775+0000","last_became_peered":"2023-10-05T19:45:47.888775+0000","last_unstale":"2023-10-05T19:45:47.892875+0000","last_undegraded":"2023-10-05T19:45:47.892875+0000","last_fullsized":"2023-10-05T19:45:47.892875+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:59:43.012571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.887113+0000","last_change":"2023-10-05T19:45:47.884687+0000","last_active":"2023-10-05T19:45:47.887113+0000","last_peered":"2023-10-05T19:45:47.887113+0000","last_clean":"2023-10-05T19:45:47.887113+0000","last_became_active":"2023-10-05T19:45:47.884436+0000","last_became_peered":"2023-10-05T19:45:47.884436+0000","last_unstale":"2023-10-05T19:45:47.887113+0000","last_undegraded":"2023-10-05T19:45:47.887113+0000","last_fullsized":"2023-10-05T19:45:47.887113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:23:53.691689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888730+0000","last_change":"2023-10-05T19:45:47.887674+0000","last_active":"2023-10-05T19:45:47.888730+0000","last_peered":"2023-10-05T19:45:47.888730+0000","last_clean":"2023-10-05T19:45:47.888730+0000","last_became_active":"2023-10-05T19:45:47.887439+0000","last_became_peered":"2023-10-05T19:45:47.887439+0000","last_unstale":"2023-10-05T19:45:47.888730+0000","last_undegraded":"2023-10-05T19:45:47.888730+0000","last_fullsized":"2023-10-05T19:45:47.888730+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:50:22.419413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297376,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560224,"statfs":{"total":107374182400,"available":107069669376,"internally_reserved":0,"allocated":90112,"data_stored":22456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298212,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559388,"statfs":{"total":107374182400,"available":107068813312,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298272,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559328,"statfs":{"total":107374182400,"available":107068751872,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:45:52.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:45:52.956 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:52.957 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2023-10-05T19:45:52.968 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-05T19:45:51.874255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893312,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679488,"statfs":{"total":322122547200,"available":321207795712,"internally_reserved":0,"allocated":1482752,"data_stored":1259612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000481"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.889998+0000","last_change":"2023-10-05T19:45:47.888567+0000","last_active":"2023-10-05T19:45:47.889998+0000","last_peered":"2023-10-05T19:45:47.889998+0000","last_clean":"2023-10-05T19:45:47.889998+0000","last_became_active":"2023-10-05T19:45:47.888343+0000","last_became_peered":"2023-10-05T19:45:47.888343+0000","last_unstale":"2023-10-05T19:45:47.889998+0000","last_undegraded":"2023-10-05T19:45:47.889998+0000","last_fullsized":"2023-10-05T19:45:47.889998+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:49.936903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888786+0000","last_change":"2023-10-05T19:45:47.887584+0000","last_active":"2023-10-05T19:45:47.888786+0000","last_peered":"2023-10-05T19:45:47.888786+0000","last_clean":"2023-10-05T19:45:47.888786+0000","last_became_active":"2023-10-05T19:45:47.887352+0000","last_became_peered":"2023-10-05T19:45:47.887352+0000","last_unstale":"2023-10-05T19:45:47.888786+0000","last_undegraded":"2023-10-05T19:45:47.888786+0000","last_fullsized":"2023-10-05T19:45:47.888786+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:09:31.632762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888783+0000","last_change":"2023-10-05T19:45:47.887688+0000","last_active":"2023-10-05T19:45:47.888783+0000","last_peered":"2023-10-05T19:45:47.888783+0000","last_clean":"2023-10-05T19:45:47.888783+0000","last_became_active":"2023-10-05T19:45:47.887430+0000","last_became_peered":"2023-10-05T19:45:47.887430+0000","last_unstale":"2023-10-05T19:45:47.888783+0000","last_undegraded":"2023-10-05T19:45:47.888783+0000","last_fullsized":"2023-10-05T19:45:47.888783+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:44:09.989557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.892882+0000","last_change":"2023-10-05T19:45:47.889252+0000","last_active":"2023-10-05T19:45:47.892882+0000","last_peered":"2023-10-05T19:45:47.892882+0000","last_clean":"2023-10-05T19:45:47.892882+0000","last_became_active":"2023-10-05T19:45:47.889058+0000","last_became_peered":"2023-10-05T19:45:47.889058+0000","last_unstale":"2023-10-05T19:45:47.892882+0000","last_undegraded":"2023-10-05T19:45:47.892882+0000","last_fullsized":"2023-10-05T19:45:47.892882+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:03:39.173092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:48.367886+0000","last_change":"2023-10-05T19:45:47.888051+0000","last_active":"2023-10-05T19:45:48.367886+0000","last_peered":"2023-10-05T19:45:48.367886+0000","last_clean":"2023-10-05T19:45:48.367886+0000","last_became_active":"2023-10-05T19:45:47.887338+0000","last_became_peered":"2023-10-05T19:45:47.887338+0000","last_unstale":"2023-10-05T19:45:48.367886+0000","last_undegraded":"2023-10-05T19:45:48.367886+0000","last_fullsized":"2023-10-05T19:45:48.367886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:18:45.531480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.889985+0000","last_change":"2023-10-05T19:45:47.888336+0000","last_active":"2023-10-05T19:45:47.889985+0000","last_peered":"2023-10-05T19:45:47.889985+0000","last_clean":"2023-10-05T19:45:47.889985+0000","last_became_active":"2023-10-05T19:45:47.887988+0000","last_became_peered":"2023-10-05T19:45:47.887988+0000","last_unstale":"2023-10-05T19:45:47.889985+0000","last_undegraded":"2023-10-05T19:45:47.889985+0000","last_fullsized":"2023-10-05T19:45:47.889985+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:50:58.570285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.892875+0000","last_change":"2023-10-05T19:45:47.888950+0000","last_active":"2023-10-05T19:45:47.892875+0000","last_peered":"2023-10-05T19:45:47.892875+0000","last_clean":"2023-10-05T19:45:47.892875+0000","last_became_active":"2023-10-05T19:45:47.888775+0000","last_became_peered":"2023-10-05T19:45:47.888775+0000","last_unstale":"2023-10-05T19:45:47.892875+0000","last_undegraded":"2023-10-05T19:45:47.892875+0000","last_fullsized":"2023-10-05T19:45:47.892875+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:59:43.012571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.887113+0000","last_change":"2023-10-05T19:45:47.884687+0000","last_active":"2023-10-05T19:45:47.887113+0000","last_peered":"2023-10-05T19:45:47.887113+0000","last_clean":"2023-10-05T19:45:47.887113+0000","last_became_active":"2023-10-05T19:45:47.884436+0000","last_became_peered":"2023-10-05T19:45:47.884436+0000","last_unstale":"2023-10-05T19:45:47.887113+0000","last_undegraded":"2023-10-05T19:45:47.887113+0000","last_fullsized":"2023-10-05T19:45:47.887113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:23:53.691689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:45:47.888730+0000","last_change":"2023-10-05T19:45:47.887674+0000","last_active":"2023-10-05T19:45:47.888730+0000","last_peered":"2023-10-05T19:45:47.888730+0000","last_clean":"2023-10-05T19:45:47.888730+0000","last_became_active":"2023-10-05T19:45:47.887439+0000","last_became_peered":"2023-10-05T19:45:47.887439+0000","last_unstale":"2023-10-05T19:45:47.888730+0000","last_undegraded":"2023-10-05T19:45:47.888730+0000","last_fullsized":"2023-10-05T19:45:47.888730+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:45:46.875962+0000","last_clean_scrub_stamp":"2023-10-05T19:45:46.875962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:50:22.419413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297384,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560216,"statfs":{"total":107374182400,"available":107069661184,"internally_reserved":0,"allocated":98304,"data_stored":26292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297964,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559636,"statfs":{"total":107374182400,"available":107069067264,"internally_reserved":0,"allocated":692224,"data_stored":616660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297964,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559636,"statfs":{"total":107374182400,"available":107069067264,"internally_reserved":0,"allocated":692224,"data_stored":616660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:45:52.969 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T19:45:52.970 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T19:45:52.970 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T19:45:52.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T19:45:53.314 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:45:53.314 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T19:45:53.328 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T19:45:53.328 INFO:teuthology.run_tasks:Running task vault... 2023-10-05T19:45:53.340 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'kv', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/kv/data/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1', 'secret': 'a2V5MS5GcWVxKzhzTGNLaGtzQkg5NGVpb1FKcFpGb2c='}, {'path': 'my-key-2', 'secret': 'a2V5Mi5yNUNNMGFzMVdIUVZxcCt5NGVmVGlQQ1k4YWg='}]}} 2023-10-05T19:45:53.340 INFO:tasks.vault:Downloading Vault... 2023-10-05T19:45:53.340 INFO:tasks.vault:Downloading Vault... 2023-10-05T19:45:53.341 DEBUG:teuthology.orchestra.run.smithi172:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2023-10-05T19:45:53.372 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:45:53.372 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:45:53.884 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 31 42.6M 31 13.3M 0 0 56.3M 0 --:--:-- --:--:-- --:--:-- 56.1M 100 42.6M 100 42.6M 0 0 83.1M 0 --:--:-- --:--:-- --:--:-- 83.0M 2023-10-05T19:45:53.886 INFO:tasks.vault:Verifying SHA256 signature... 2023-10-05T19:45:53.886 DEBUG:teuthology.orchestra.run.smithi172:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2023-10-05T19:45:54.051 INFO:tasks.vault:Extracting vault... 2023-10-05T19:45:54.051 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p /home/ubuntu/cephtest/vault 2023-10-05T19:45:54.065 DEBUG:teuthology.orchestra.run.smithi172:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2023-10-05T19:45:54.125 INFO:teuthology.orchestra.run.smithi172.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2023-10-05T19:45:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2023-10-05T19:45:54.849 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2023-10-05T19:45:54.850 INFO:tasks.vault.client.0:Restarting daemon 2023-10-05T19:45:54.850 DEBUG:teuthology.orchestra.run.smithi172:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2023-10-05T19:45:54.892 INFO:tasks.vault.client.0:Started 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout:==> Vault server configuration: 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Api Address: http://0.0.0.0:8200 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Cgo: disabled 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Cluster Address: https://0.0.0.0:8201 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Log Level: info 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Mlock: supported: true, enabled: false 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Storage: inmem 2023-10-05T19:45:54.936 INFO:tasks.vault.client.0.smithi172.stdout: Version: Vault v1.2.2 2023-10-05T19:45:54.937 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout:and starts unsealed with a single unseal key. The root token is already 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout:You may need to set the following environment variable: 2023-10-05T19:45:54.953 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:The unseal key and root token are displayed below in case you want to 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:seal/unseal the Vault or re-authenticate. 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:Unseal Key: 6FEZJt2dpbUSmKddygk0Ax9Yucw/Vz/Sepv2AkgCyTc= 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:Root Token: test_root_token 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:Development mode should NOT be used in production installations! 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout:==> Vault server started! Log data will stream in below: 2023-10-05T19:45:54.954 INFO:tasks.vault.client.0.smithi172.stdout: 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.931Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.931Z [INFO] core: security barrier not initialized 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.931Z [INFO] core: security barrier initialized: shares=1 threshold=1 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.932Z [INFO] core: post-unseal setup starting 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.940Z [INFO] core: loaded wrapping token key 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.940Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T19:45:54.955 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.940Z [INFO] core: no mounts; adding default mount table 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.941Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.941Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.941Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] core: restoring leases 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] rollback: starting rollback manager 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] expiration: lease restore complete 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] identity: entities restored 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] identity: groups restored 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.942Z [INFO] core: post-unseal setup complete 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: root token generated 2023-10-05T19:45:54.956 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: pre-seal teardown starting 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] rollback: stopping rollback manager 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: pre-seal teardown complete 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: post-unseal setup starting 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: loaded wrapping token key 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.943Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T19:45:54.957 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] core: restoring leases 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] rollback: starting rollback manager 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] identity: entities restored 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.944Z [INFO] identity: groups restored 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.945Z [INFO] core: post-unseal setup complete 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.945Z [INFO] core: vault is unsealed 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.945Z [INFO] expiration: lease restore complete 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.946Z [INFO] expiration: revoked lease: lease_id=auth/token/root/h8d63df8184152a53ae34d97c9a1447434ca783d02baa9f3eed03e64113fea4a7 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.947Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2023-10-05T19:45:54.958 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.957Z [INFO] secrets.kv.kv_7d727b43: collecting keys to upgrade 2023-10-05T19:45:54.959 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.957Z [INFO] secrets.kv.kv_7d727b43: done collecting keys: num_keys=1 2023-10-05T19:45:54.959 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:45:54.957Z [INFO] secrets.kv.kv_7d727b43: upgrading keys finished 2023-10-05T19:46:04.893 INFO:tasks.vault:Mounting kv version 2 secrets engine 2023-10-05T19:46:04.893 INFO:tasks.vault:Send request to Vault: smithi172.front.sepia.ceph.com:8200 at /v1/sys/mounts/kv with token: test_root_token 2023-10-05T19:46:04.900 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:46:04.899Z [INFO] core: successful mount: namespace= path=kv/ type=kv 2023-10-05T19:46:04.901 INFO:tasks.vault:b'' 2023-10-05T19:46:04.901 INFO:tasks.vault:Send request to Vault: smithi172.front.sepia.ceph.com:8200 at /v1/kv/data/my-key-1 with token: test_root_token 2023-10-05T19:46:04.910 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:46:04.909Z [INFO] secrets.kv.kv_eece86d8: collecting keys to upgrade 2023-10-05T19:46:04.910 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:46:04.909Z [INFO] secrets.kv.kv_eece86d8: done collecting keys: num_keys=3 2023-10-05T19:46:04.910 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:46:04.909Z [INFO] secrets.kv.kv_eece86d8: upgrading keys finished 2023-10-05T19:46:04.919 INFO:tasks.vault:b'{"request_id":"84e4aae2-fe68-41ac-0026-63e9d115e49a","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2023-10-05T19:46:04.91838068Z","deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}\n' 2023-10-05T19:46:04.919 INFO:tasks.vault:Send request to Vault: smithi172.front.sepia.ceph.com:8200 at /v1/kv/data/my-key-2 with token: test_root_token 2023-10-05T19:46:04.921 INFO:tasks.vault:b'{"request_id":"5bb541ba-24a9-0b78-ba13-0da671701905","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2023-10-05T19:46:04.921216425Z","deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}\n' 2023-10-05T19:46:04.922 INFO:tasks.vault:secrets created 2023-10-05T19:46:04.922 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T19:46:04.935 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2023-10-05T19:46:04.935 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T19:46:04.935 INFO:tasks.rgw:Creating data pools 2023-10-05T19:46:04.935 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T19:46:04.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-10-05T19:46:05.985 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T19:46:06.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T19:46:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T19:46:07.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T19:46:07.998 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T19:46:08.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T19:46:09.015 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T19:46:09.026 DEBUG:tasks.rgw:Pools created 2023-10-05T19:46:09.026 INFO:tasks.rgw:Starting rgw... 2023-10-05T19:46:09.027 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2023-10-05T19:46:09.027 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T19:46:09.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2023-10-05T19:46:09.053 INFO:teuthology.orchestra.run.smithi172.stdout:test_root_token 2023-10-05T19:46:09.054 INFO:tasks.rgw:Token file content 2023-10-05T19:46:09.054 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/ceph/vault-root-token 2023-10-05T19:46:09.108 INFO:teuthology.orchestra.run.smithi172.stdout:test_root_token 2023-10-05T19:46:09.109 INFO:tasks.rgw:Restrict access to token file 2023-10-05T19:46:09.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 600 /etc/ceph/vault-root-token 2023-10-05T19:46:09.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown ceph /etc/ceph/vault-root-token 2023-10-05T19:46:09.237 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T19:46:09.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi172.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi172.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T19:46:09.279 INFO:tasks.rgw.client.0:Started 2023-10-05T19:46:09.279 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:09.279 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:09.297 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:46:09.297 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:46:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:46:09.298 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:46:10.299 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:10.318 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:46:10.318 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:46:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:46:10.320 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:46:12.321 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:12.343 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:46:12.343 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:46:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:46:12.345 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:46:16.346 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:16.366 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:46:16.366 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:46:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:46:16.369 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi172.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:46:24.370 DEBUG:teuthology.orchestra.run.smithi172:> curl http://smithi172.front.sepia.ceph.com:80/ 2023-10-05T19:46:24.393 INFO:teuthology.orchestra.run.smithi172.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:46:24.393 INFO:teuthology.orchestra.run.smithi172.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:46:24.394 INFO:teuthology.orchestra.run.smithi172.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2023-10-05T19:46:24.395 INFO:teuthology.orchestra.run.smithi172.stdout:anonymous 2023-10-05T19:46:24.395 INFO:teuthology.run_tasks:Running task s3tests... 2023-10-05T19:46:24.412 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-10-05T19:46:24.412 INFO:tasks.s3tests:Downloading s3-tests... 2023-10-05T19:46:24.412 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-10-05T19:46:24.412 DEBUG:teuthology.orchestra.run.smithi172:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:46:24.471 INFO:teuthology.orchestra.run.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2023-10-05T19:46:25.642 INFO:tasks.s3tests:Creating rgw users... 2023-10-05T19:46:25.643 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-10-05T19:46:25.643 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key CYBQAIVZPJQSUWJJEDQA --secret A7ouGX1KtwCs5JJ+U7Ea2AstZCus1sKckDqxy2l68qQrG/hp2cRnhg== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:46:25.735 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:25.735 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:25.741 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.739+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 20 realm 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.740+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.741+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.743 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.741+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.741+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.741+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.742+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:25.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.742+0000 7fd711e9b680 20 searching for the correct realm 2023-10-05T19:46:25.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.749+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.752 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.750+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:25.759 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.757+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:25.760 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 zone default found 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 4 Realm: () 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 zonegroup default 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.758+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.759+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.759+0000 7fd711e9b680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.759+0000 7fd711e9b680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.759+0000 7fd711e9b680 20 started sync module instance, tier type = 2023-10-05T19:46:25.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.759+0000 7fd711e9b680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=0 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=1 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=2 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=3 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=4 2023-10-05T19:46:25.785 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=5 2023-10-05T19:46:25.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.783+0000 7fd711e9b680 20 add_watcher() i=6 2023-10-05T19:46:25.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.784+0000 7fd711e9b680 20 add_watcher() i=7 2023-10-05T19:46:25.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.784+0000 7fd711e9b680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:25.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.785+0000 7fd6d67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:25.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.785+0000 7fd711e9b680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:25.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.785+0000 7fd711e9b680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:25.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.789+0000 7fd711e9b680 5 note: GC not initialized 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd6457fa700 20 reqs_thread_entry: start 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd63effd700 20 reqs_thread_entry: start 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd711e9b680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd711e9b680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd63d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:56 2023 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.790+0000 7fd63cff9700 20 reqs_thread_entry: start 2023-10-05T19:46:25.794 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 20 get_system_obj_state: rctx=0x7ffd46431ae8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5575f5c39240 s->prefetch_data=0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 20 get_system_obj_state: rctx=0x7ffd464317d8 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x5575f5c39240 s->prefetch_data=0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 20 get_system_obj_state: rctx=0x7ffd464317d8 obj=default.rgw.meta:users.keys:CYBQAIVZPJQSUWJJEDQA state=0x5575f5c39240 s->prefetch_data=0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache get: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA : miss 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA info.flags=0x0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 adding default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA to cache LRU end 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 20 get_system_obj_state: rctx=0x7ffd46431068 obj=default.rgw.meta:users.keys:CYBQAIVZPJQSUWJJEDQA state=0x5575f5c39240 s->prefetch_data=0 2023-10-05T19:46:25.795 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache get: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA : hit (negative entry) 2023-10-05T19:46:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 20 get_system_obj_state: rctx=0x7ffd46431718 obj=default.rgw.meta:users.keys:CYBQAIVZPJQSUWJJEDQA state=0x5575f5c39240 s->prefetch_data=0 2023-10-05T19:46:25.796 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.791+0000 7fd711e9b680 10 cache get: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA : hit (negative entry) 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.792+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.792+0000 7fd711e9b680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.792+0000 7fd711e9b680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.793+0000 7fd6feffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215104 cookie 93965124381072 notifier 4210 bl.length()=640 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.793+0000 7fd6feffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.793+0000 7fd6feffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd711e9b680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd711e9b680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T19:46:25.797 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd6fe7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215104 cookie 93965124390800 notifier 4210 bl.length()=194 2023-10-05T19:46:25.798 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd6fe7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:25.798 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.794+0000 7fd6fe7fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:46:25.798 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd711e9b680 10 cache put: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA info.flags=0x7 2023-10-05T19:46:25.798 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd711e9b680 10 moving default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA to cache LRU end 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd711e9b680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:CYBQAIVZPJQSUWJJEDQA, ofs0, ns] 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd6feffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215105 cookie 93965124382176 notifier 4210 bl.length()=186 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd6feffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA info.flags=0x7 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.796+0000 7fd6feffd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+CYBQAIVZPJQSUWJJEDQA to cache LRU end 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "user_id": "foo.client.0", 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "display_name": "Mr. foo.client.0", 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "email": "foo.client.0+test@test.test", 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "suspended": 0, 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "max_buckets": 1000, 2023-10-05T19:46:25.799 INFO:teuthology.orchestra.run.smithi172.stdout: "subusers": [], 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "keys": [ 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "user": "foo.client.0", 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "CYBQAIVZPJQSUWJJEDQA", 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "A7ouGX1KtwCs5JJ+U7Ea2AstZCus1sKckDqxy2l68qQrG/hp2cRnhg==" 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "swift_keys": [], 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": [ 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "user-policy", 2023-10-05T19:46:25.800 INFO:teuthology.orchestra.run.smithi172.stdout: "perm": "*" 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "op_mask": "read, write, delete", 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "", 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "default_storage_class": "", 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_tags": [], 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:25.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "temp_url_keys": [], 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "rgw", 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: "mfa_ids": [] 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-05T19:46:25.802 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:25.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.798+0000 7fd711e9b680 20 remove_watcher() i=0 2023-10-05T19:46:25.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.798+0000 7fd711e9b680 2 removed watcher, disabling cache 2023-10-05T19:46:25.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.799+0000 7fd711e9b680 20 remove_watcher() i=1 2023-10-05T19:46:25.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.799+0000 7fd711e9b680 20 remove_watcher() i=2 2023-10-05T19:46:25.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.800+0000 7fd711e9b680 20 remove_watcher() i=3 2023-10-05T19:46:25.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.801+0000 7fd711e9b680 20 remove_watcher() i=4 2023-10-05T19:46:25.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.802+0000 7fd711e9b680 20 remove_watcher() i=5 2023-10-05T19:46:25.805 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.803+0000 7fd711e9b680 20 remove_watcher() i=6 2023-10-05T19:46:25.806 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.804+0000 7fd711e9b680 20 remove_watcher() i=7 2023-10-05T19:46:25.807 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.805+0000 7fd6d67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:25.811 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 7260327535 --totp-seed P4AL6REMNCROWWXKKFTS2JGQIPPU6DYZCZDL6GCHMNV7HMFSUKSMHZGFCUTXAHNC --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:46:25.860 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:25.860 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:25.865 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.863+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.866 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 20 realm 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.864+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.865+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.867 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.865+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.865+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.865+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.865+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:25.868 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.866+0000 7f5d17e10680 20 searching for the correct realm 2023-10-05T19:46:25.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.872+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:25.875 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.872+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.872+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.872+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.872+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.873+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.876 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.874+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.877 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.874+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.877 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.874+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.880+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.880+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.880+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.880+0000 7f5d17e10680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.880+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.883 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.881+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 zone default found 2023-10-05T19:46:25.885 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 4 Realm: () 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 zonegroup default 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 started sync module instance, tier type = 2023-10-05T19:46:25.886 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.882+0000 7f5d17e10680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=0 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=1 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=2 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=3 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=4 2023-10-05T19:46:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=5 2023-10-05T19:46:25.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.891+0000 7f5d17e10680 20 add_watcher() i=6 2023-10-05T19:46:25.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.892+0000 7f5d17e10680 20 add_watcher() i=7 2023-10-05T19:46:25.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.892+0000 7f5d17e10680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:25.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.893+0000 7f5cdbfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:25.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.893+0000 7f5d17e10680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:25.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.893+0000 7f5d17e10680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:25.899 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.897+0000 7f5d17e10680 5 note: GC not initialized 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5c4affd700 20 reqs_thread_entry: start 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5c48ff9700 20 reqs_thread_entry: start 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5c3affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:56 2023 2023-10-05T19:46:25.900 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:25.901 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5c3a7fc700 20 reqs_thread_entry: start 2023-10-05T19:46:25.901 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 20 get_system_obj_state: rctx=0x7fff78919a88 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557ad809e240 s->prefetch_data=0 2023-10-05T19:46:25.901 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:46:25.901 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 20 Read xattr: user.rgw.idtag 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.898+0000 7f5d17e10680 20 rados->read ofs=0 len=0 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.899+0000 7f5d17e10680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.899+0000 7f5d17e10680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.899+0000 7f5d17e10680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:25.902 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:25.899+0000 7f5d17e10680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:27.076 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5d17e10680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:46:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5d17e10680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5d17e10680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:46:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5cfbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 93986100495840 notifier 4216 bl.length()=654 2023-10-05T19:46:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5cfbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:46:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.074+0000 7f5cfbfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:46:27.078 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.077+0000 7f5d17e10680 20 remove_watcher() i=0 2023-10-05T19:46:27.079 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.077+0000 7f5d17e10680 2 removed watcher, disabling cache 2023-10-05T19:46:27.079 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.078+0000 7f5d17e10680 20 remove_watcher() i=1 2023-10-05T19:46:27.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.080+0000 7f5d17e10680 20 remove_watcher() i=2 2023-10-05T19:46:27.085 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.083+0000 7f5d17e10680 20 remove_watcher() i=3 2023-10-05T19:46:27.088 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.086+0000 7f5d17e10680 20 remove_watcher() i=4 2023-10-05T19:46:27.089 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.087+0000 7f5d17e10680 20 remove_watcher() i=5 2023-10-05T19:46:27.092 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.090+0000 7f5d17e10680 20 remove_watcher() i=6 2023-10-05T19:46:27.093 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.091+0000 7f5d17e10680 20 remove_watcher() i=7 2023-10-05T19:46:27.093 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.091+0000 7f5cdbfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.099 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-10-05T19:46:27.100 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key XDJQXDVKPUZOUVCAGMHV --secret zKCGmA9+Dys7i5wJi85o5KEWxgHWoyyB92YKsUvLg75Pk9kFlWEqag== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:46:27.150 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:27.151 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:27.156 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.154+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.157 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 realm 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.155+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.156+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.158 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.156+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.156+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.156+0000 7f36c5f2c680 20 searching for the correct realm 2023-10-05T19:46:27.166 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.163+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.166 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.166 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.164+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.168 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.165+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.168 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.165+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.168 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.165+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.171+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.171+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.171+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.171+0000 7f36c5f2c680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.171+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.174 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.175 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 20 zone default found 2023-10-05T19:46:27.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.172+0000 7f36c5f2c680 4 Realm: () 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 zonegroup default 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 started sync module instance, tier type = 2023-10-05T19:46:27.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.173+0000 7f36c5f2c680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:27.202 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=0 2023-10-05T19:46:27.202 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=1 2023-10-05T19:46:27.202 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=2 2023-10-05T19:46:27.202 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=3 2023-10-05T19:46:27.203 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=4 2023-10-05T19:46:27.203 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=5 2023-10-05T19:46:27.203 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.200+0000 7f36c5f2c680 20 add_watcher() i=6 2023-10-05T19:46:27.203 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.201+0000 7f36c5f2c680 20 add_watcher() i=7 2023-10-05T19:46:27.203 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.201+0000 7f36c5f2c680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:27.205 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.203+0000 7f368a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.205 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.203+0000 7f36c5f2c680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:27.205 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.203+0000 7f36c5f2c680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:27.209 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.207+0000 7f36c5f2c680 5 note: GC not initialized 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.207+0000 7f35f97fa700 20 reqs_thread_entry: start 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f35eaffd700 20 reqs_thread_entry: start 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f35e97fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:57 2023 2023-10-05T19:46:27.211 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f35e8ff9700 20 reqs_thread_entry: start 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 20 get_system_obj_state: rctx=0x7ffd0fd79c98 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55c194f60240 s->prefetch_data=0 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 20 get_system_obj_state: rctx=0x7ffd0fd79988 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55c194f60240 s->prefetch_data=0 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.208+0000 7f36c5f2c680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 20 get_system_obj_state: rctx=0x7ffd0fd79988 obj=default.rgw.meta:users.keys:XDJQXDVKPUZOUVCAGMHV state=0x55c194f60240 s->prefetch_data=0 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 cache get: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV : miss 2023-10-05T19:46:27.212 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV info.flags=0x0 2023-10-05T19:46:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 adding default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV to cache LRU end 2023-10-05T19:46:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 20 get_system_obj_state: rctx=0x7ffd0fd79218 obj=default.rgw.meta:users.keys:XDJQXDVKPUZOUVCAGMHV state=0x55c194f60240 s->prefetch_data=0 2023-10-05T19:46:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 cache get: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV : hit (negative entry) 2023-10-05T19:46:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 20 get_system_obj_state: rctx=0x7ffd0fd798c8 obj=default.rgw.meta:users.keys:XDJQXDVKPUZOUVCAGMHV state=0x55c194f60240 s->prefetch_data=0 2023-10-05T19:46:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.209+0000 7f36c5f2c680 10 cache get: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV : hit (negative entry) 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36c5f2c680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36c5f2c680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36aaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149698 cookie 94289917813424 notifier 4222 bl.length()=640 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36aaffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:46:27.214 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.210+0000 7f36aaffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36c5f2c680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36c5f2c680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36aa7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149699 cookie 94289917813424 notifier 4222 bl.length()=194 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36aa7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.212+0000 7f36aa7fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.216 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.214+0000 7f36c5f2c680 10 cache put: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV info.flags=0x7 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.214+0000 7f36c5f2c680 10 moving default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV to cache LRU end 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.214+0000 7f36c5f2c680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:XDJQXDVKPUZOUVCAGMHV, ofs0, ns] 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.215+0000 7f36aaffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149698 cookie 94289917823312 notifier 4222 bl.length()=186 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.215+0000 7f36aaffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV info.flags=0x7 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.215+0000 7f36aaffd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+XDJQXDVKPUZOUVCAGMHV to cache LRU end 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-05T19:46:27.217 INFO:teuthology.orchestra.run.smithi172.stdout: "user_id": "bar.client.0", 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "display_name": "Mr. bar.client.0", 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "email": "bar.client.0+test@test.test", 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "suspended": 0, 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "max_buckets": 1000, 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "subusers": [], 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "keys": [ 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "user": "bar.client.0", 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "XDJQXDVKPUZOUVCAGMHV", 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "zKCGmA9+Dys7i5wJi85o5KEWxgHWoyyB92YKsUvLg75Pk9kFlWEqag==" 2023-10-05T19:46:27.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "swift_keys": [], 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": [ 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "user-policy", 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "perm": "*" 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "op_mask": "read, write, delete", 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "", 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "default_storage_class": "", 2023-10-05T19:46:27.219 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_tags": [], 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:27.220 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout: "temp_url_keys": [], 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "rgw", 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout: "mfa_ids": [] 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:27.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.216+0000 7f36c5f2c680 20 remove_watcher() i=0 2023-10-05T19:46:27.222 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.216+0000 7f36c5f2c680 2 removed watcher, disabling cache 2023-10-05T19:46:27.222 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.219+0000 7f36c5f2c680 20 remove_watcher() i=1 2023-10-05T19:46:27.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.221+0000 7f36c5f2c680 20 remove_watcher() i=2 2023-10-05T19:46:27.226 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.224+0000 7f36c5f2c680 20 remove_watcher() i=3 2023-10-05T19:46:27.229 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.227+0000 7f36c5f2c680 20 remove_watcher() i=4 2023-10-05T19:46:27.230 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.228+0000 7f36c5f2c680 20 remove_watcher() i=5 2023-10-05T19:46:27.234 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.232+0000 7f36c5f2c680 20 remove_watcher() i=6 2023-10-05T19:46:27.235 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.233+0000 7f36c5f2c680 20 remove_watcher() i=7 2023-10-05T19:46:27.239 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.234+0000 7f368a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.241 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 8581428241 --totp-seed VBGKSYA7RATY2VASPNOG74G23KPXLLW7VQYGLMH6NR2AYTLNIBD7RPYLARWVI3MK --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:46:27.289 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:27.289 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:27.295 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.293+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 realm 2023-10-05T19:46:27.296 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.294+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.298 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.295+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.298 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.295+0000 7f593550d680 20 searching for the correct realm 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.305 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.302+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.306 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.303+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.313 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.309+0000 7f593550d680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.313 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.309+0000 7f593550d680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.313 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.309+0000 7f593550d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.313 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.309+0000 7f593550d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.313 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.309+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.310+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 zone default found 2023-10-05T19:46:27.314 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 4 Realm: () 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 zonegroup default 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 started sync module instance, tier type = 2023-10-05T19:46:27.315 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.311+0000 7f593550d680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=0 2023-10-05T19:46:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=1 2023-10-05T19:46:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=2 2023-10-05T19:46:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=3 2023-10-05T19:46:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=4 2023-10-05T19:46:27.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=5 2023-10-05T19:46:27.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.322+0000 7f593550d680 20 add_watcher() i=6 2023-10-05T19:46:27.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.323+0000 7f593550d680 20 add_watcher() i=7 2023-10-05T19:46:27.325 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.323+0000 7f593550d680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:27.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.324+0000 7f58f9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.324+0000 7f593550d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:27.326 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.324+0000 7f593550d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:27.330 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.328+0000 7f593550d680 5 note: GC not initialized 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f5870ff9700 20 reqs_thread_entry: start 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f58667fc700 20 reqs_thread_entry: start 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f593550d680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f593550d680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f5864ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:57 2023 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.329+0000 7f584bfff700 20 reqs_thread_entry: start 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 20 get_system_obj_state: rctx=0x7ffeb724df08 obj=default.rgw.meta:users.uid:bar.client.0 state=0x562242caa240 s->prefetch_data=0 2023-10-05T19:46:27.332 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 20 Read xattr: user.rgw.idtag 2023-10-05T19:46:27.333 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.330+0000 7f593550d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.332+0000 7f593550d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.332+0000 7f593550d680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.334 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.332+0000 7f593550d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:46:27.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.333+0000 7f5926602700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149700 cookie 94705151066688 notifier 4228 bl.length()=654 2023-10-05T19:46:27.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.333+0000 7f5926602700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:46:27.335 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.333+0000 7f5926602700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:46:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.334+0000 7f593550d680 20 remove_watcher() i=0 2023-10-05T19:46:27.336 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.334+0000 7f593550d680 2 removed watcher, disabling cache 2023-10-05T19:46:27.337 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.335+0000 7f593550d680 20 remove_watcher() i=1 2023-10-05T19:46:27.338 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.336+0000 7f593550d680 20 remove_watcher() i=2 2023-10-05T19:46:27.339 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.337+0000 7f593550d680 20 remove_watcher() i=3 2023-10-05T19:46:27.340 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.338+0000 7f593550d680 20 remove_watcher() i=4 2023-10-05T19:46:27.341 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.339+0000 7f593550d680 20 remove_watcher() i=5 2023-10-05T19:46:27.342 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.340+0000 7f593550d680 20 remove_watcher() i=6 2023-10-05T19:46:27.343 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.341+0000 7f593550d680 20 remove_watcher() i=7 2023-10-05T19:46:27.343 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.341+0000 7f58f9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.348 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-10-05T19:46:27.348 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key IPLOTLGSDBZYLVCRYQHI --secret 7HNiSD5AMzI79TnvO58sq/b8REAAnKwIu1Xptjkdr5eBhOra1iL+0Q== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-10-05T19:46:27.397 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:27.400 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:27.402 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.400+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.401+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.401+0000 7f6d3ff18680 20 realm 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.401+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.402+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.403+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.403+0000 7f6d3ff18680 20 searching for the correct realm 2023-10-05T19:46:27.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.413 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.410+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.411+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.411+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.411+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.411+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.414 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.411+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.417+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 zone default found 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 4 Realm: () 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 zonegroup default 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.418+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 started sync module instance, tier type = 2023-10-05T19:46:27.424 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.419+0000 7f6d3ff18680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=0 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=1 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=2 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=3 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=4 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=5 2023-10-05T19:46:27.443 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.440+0000 7f6d3ff18680 20 add_watcher() i=6 2023-10-05T19:46:27.445 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.443+0000 7f6d3ff18680 20 add_watcher() i=7 2023-10-05T19:46:27.445 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.443+0000 7f6d3ff18680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:27.446 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.444+0000 7f6d04fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.447 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.445+0000 7f6d3ff18680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:27.447 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.445+0000 7f6d3ff18680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:27.450 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.448+0000 7f6d3ff18680 5 note: GC not initialized 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.449+0000 7f6c737fe700 20 reqs_thread_entry: start 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.449+0000 7f6c717fa700 20 reqs_thread_entry: start 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.449+0000 7f6d3ff18680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6c5f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:57 2023 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6c5effd700 20 reqs_thread_entry: start 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 20 get_system_obj_state: rctx=0x7ffe63635d98 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5651cdc5b240 s->prefetch_data=0 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:46:27.453 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 20 get_system_obj_state: rctx=0x7ffe63635a88 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x5651cdc5b240 s->prefetch_data=0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 20 get_system_obj_state: rctx=0x7ffe63635a88 obj=default.rgw.meta:users.keys:IPLOTLGSDBZYLVCRYQHI state=0x5651cdc5b240 s->prefetch_data=0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.450+0000 7f6d3ff18680 10 cache get: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI : miss 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI info.flags=0x0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 10 adding default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI to cache LRU end 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 20 get_system_obj_state: rctx=0x7ffe63635318 obj=default.rgw.meta:users.keys:IPLOTLGSDBZYLVCRYQHI state=0x5651cdc5b240 s->prefetch_data=0 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 10 cache get: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI : hit (negative entry) 2023-10-05T19:46:27.454 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 20 get_system_obj_state: rctx=0x7ffe636359c8 obj=default.rgw.meta:users.keys:IPLOTLGSDBZYLVCRYQHI state=0x5651cdc5b240 s->prefetch_data=0 2023-10-05T19:46:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.451+0000 7f6d3ff18680 10 cache get: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI : hit (negative entry) 2023-10-05T19:46:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:46:27.455 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3ff18680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3ff18680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:46:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3100d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149699 cookie 94909346255360 notifier 4234 bl.length()=714 2023-10-05T19:46:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3100d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:46:27.456 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.452+0000 7f6d3100d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3ff18680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3ff18680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3080c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149700 cookie 94909346250208 notifier 4234 bl.length()=224 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3080c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:46:27.458 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.456+0000 7f6d3080c700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:46:27.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.457+0000 7f6d3ff18680 10 cache put: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI info.flags=0x7 2023-10-05T19:46:27.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.457+0000 7f6d3ff18680 10 moving default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI to cache LRU end 2023-10-05T19:46:27.459 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.457+0000 7f6d3ff18680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:IPLOTLGSDBZYLVCRYQHI, ofs0, ns] 2023-10-05T19:46:27.460 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.458+0000 7f6d3100d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 94909346244512 notifier 4234 bl.length()=201 2023-10-05T19:46:27.460 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.458+0000 7f6d3100d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI info.flags=0x7 2023-10-05T19:46:27.460 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.458+0000 7f6d3100d700 10 rgw watcher librados: moving default.rgw.meta+users.keys+IPLOTLGSDBZYLVCRYQHI to cache LRU end 2023-10-05T19:46:27.460 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "user_id": "testx$tenanteduser.client.0", 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "suspended": 0, 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "max_buckets": 1000, 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "subusers": [], 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "keys": [ 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "user": "testx$tenanteduser.client.0", 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "IPLOTLGSDBZYLVCRYQHI", 2023-10-05T19:46:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "7HNiSD5AMzI79TnvO58sq/b8REAAnKwIu1Xptjkdr5eBhOra1iL+0Q==" 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "swift_keys": [], 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "caps": [ 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "user-policy", 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "perm": "*" 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "op_mask": "read, write, delete", 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "", 2023-10-05T19:46:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "default_storage_class": "", 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_tags": [], 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2023-10-05T19:46:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: "temp_url_keys": [], 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: "type": "rgw", 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: "mfa_ids": [] 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.460+0000 7f6d3ff18680 20 remove_watcher() i=0 2023-10-05T19:46:27.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.460+0000 7f6d3ff18680 2 removed watcher, disabling cache 2023-10-05T19:46:27.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.463+0000 7f6d3ff18680 20 remove_watcher() i=1 2023-10-05T19:46:27.466 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.464+0000 7f6d3ff18680 20 remove_watcher() i=2 2023-10-05T19:46:27.469 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.467+0000 7f6d3ff18680 20 remove_watcher() i=3 2023-10-05T19:46:27.471 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.469+0000 7f6d3ff18680 20 remove_watcher() i=4 2023-10-05T19:46:27.472 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.470+0000 7f6d3ff18680 20 remove_watcher() i=5 2023-10-05T19:46:27.475 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.473+0000 7f6d3ff18680 20 remove_watcher() i=6 2023-10-05T19:46:27.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.474+0000 7f6d3ff18680 20 remove_watcher() i=7 2023-10-05T19:46:27.476 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.475+0000 7f6d04fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.483 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 1690959926 --totp-seed B44VMXMT7F5FIQ266FQWMJCZJWDKHCAXFO7GGQW5EKXX76M4PVLNVK5ZANLUFKSW --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:46:27.531 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:46:27.531 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:46:27.536 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.534+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.535+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.535+0000 7f35a4466680 20 realm 2023-10-05T19:46:27.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.535+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.536+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.539 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.537+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.537+0000 7f35a4466680 20 searching for the correct realm 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.547 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.544+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.545+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.545+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.548 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.545+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.551+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:46:27.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 zone default found 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 4 Realm: () 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 zonegroup default 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.556 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.552+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.553+0000 7f35a4466680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.553+0000 7f35a4466680 20 started sync module instance, tier type = 2023-10-05T19:46:27.557 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.553+0000 7f35a4466680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:46:27.583 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=0 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=1 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=2 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=3 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=4 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=5 2023-10-05T19:46:27.584 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.581+0000 7f35a4466680 20 add_watcher() i=6 2023-10-05T19:46:27.586 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.584+0000 7f35a4466680 20 add_watcher() i=7 2023-10-05T19:46:27.586 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.584+0000 7f35a4466680 2 all 8 watchers are set, enabling cache 2023-10-05T19:46:27.589 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.587+0000 7f3568fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.589 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.587+0000 7f35a4466680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:46:27.589 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.587+0000 7f35a4466680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:46:27.592 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.590+0000 7f35a4466680 5 note: GC not initialized 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.591+0000 7f34d77fe700 20 reqs_thread_entry: start 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f34d57fa700 20 reqs_thread_entry: start 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 20 init_complete bucket index max shards: 11 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f34c37fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:46:57 2023 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:46:27.594 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f34c2ffd700 20 reqs_thread_entry: start 2023-10-05T19:46:27.595 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 20 get_system_obj_state: rctx=0x7ffe343f1668 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x565011108240 s->prefetch_data=0 2023-10-05T19:46:27.595 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 20 Read xattr: user.rgw.idtag 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.592+0000 7f35a4466680 20 rados->read ofs=0 len=0 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.593+0000 7f35a4466680 20 rados_obj.operate() r=0 bl.length=514 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.593+0000 7f35a4466680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.593+0000 7f35a4466680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.596 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.593+0000 7f35a4466680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:46:27.599 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f35a4466680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:46:27.599 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f35a4466680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.599 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f35a4466680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:46:27.599 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f3595d5c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149701 cookie 94901885288656 notifier 4240 bl.length()=728 2023-10-05T19:46:27.599 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f3595d5c700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:46:27.600 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.597+0000 7f3595d5c700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:46:27.601 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.599+0000 7f35a4466680 20 remove_watcher() i=0 2023-10-05T19:46:27.601 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.599+0000 7f35a4466680 2 removed watcher, disabling cache 2023-10-05T19:46:27.604 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.602+0000 7f35a4466680 20 remove_watcher() i=1 2023-10-05T19:46:27.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.607+0000 7f35a4466680 20 remove_watcher() i=2 2023-10-05T19:46:27.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.608+0000 7f35a4466680 20 remove_watcher() i=3 2023-10-05T19:46:27.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.610+0000 7f35a4466680 20 remove_watcher() i=4 2023-10-05T19:46:27.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.611+0000 7f35a4466680 20 remove_watcher() i=5 2023-10-05T19:46:27.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.614+0000 7f35a4466680 20 remove_watcher() i=6 2023-10-05T19:46:27.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.616+0000 7f35a4466680 20 remove_watcher() i=7 2023-10-05T19:46:27.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:46:27.617+0000 7f3568fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:46:27.624 INFO:tasks.s3tests:Configuring s3-tests... 2023-10-05T19:46:27.625 DEBUG:teuthology.orchestra.run.smithi172:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2023-10-05T19:46:34.440 INFO:teuthology.orchestra.run.smithi172.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-10-05T19:46:34.802 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-10-05T19:46:35.683 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:19 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:46:37.504 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-10-05T19:46:37.506 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:46:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-10-05T19:46:37.506 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:46:37.506 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-10-05T19:46:37.506 INFO:teuthology.orchestra.run.smithi172.stdout: libevent-devel x86_64 2.1.8-5.el8 rhel-8-for-x86_64-appstream-rpms 104 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: libffi-devel x86_64 3.1-22.el8 rhel-8-for-x86_64-baseos-rpms 29 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: libxml2-devel x86_64 2.9.7-9.el8_4.2 rhel-8-for-x86_64-appstream-rpms 1.0 M 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: libxslt-devel x86_64 1.1.32-6.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: cmake-filesystem x86_64 3.18.2-11.el8_4 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: libgcrypt-devel x86_64 1.8.5-4.el8 rhel-8-for-x86_64-baseos-rpms 149 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: libgpg-error-devel 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 1.31-1.el8 rhel-8-for-x86_64-baseos-rpms 64 k 2023-10-05T19:46:37.507 INFO:teuthology.orchestra.run.smithi172.stdout: xz-devel x86_64 5.2.4-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout:Install 9 Packages 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:37.508 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.9 M 2023-10-05T19:46:37.509 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 12 M 2023-10-05T19:46:37.509 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-10-05T19:46:37.593 INFO:teuthology.orchestra.run.smithi172.stdout:(1/9): libxml2-devel-2.9.7-9.el8_4.2.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2023-10-05T19:46:37.594 INFO:teuthology.orchestra.run.smithi172.stdout:(2/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 1.2 MB/s | 104 kB 00:00 2023-10-05T19:46:37.600 INFO:teuthology.orchestra.run.smithi172.stdout:(3/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2023-10-05T19:46:37.634 INFO:teuthology.orchestra.run.smithi172.stdout:(4/9): libffi-devel-3.1-22.el8.x86_64.rpm 719 kB/s | 29 kB 00:00 2023-10-05T19:46:37.635 INFO:teuthology.orchestra.run.smithi172.stdout:(5/9): cmake-filesystem-3.18.2-11.el8_4.x86_64. 1.0 MB/s | 44 kB 00:00 2023-10-05T19:46:37.640 INFO:teuthology.orchestra.run.smithi172.stdout:(6/9): xz-devel-5.2.4-3.el8.x86_64.rpm 1.5 MB/s | 62 kB 00:00 2023-10-05T19:46:37.665 INFO:teuthology.orchestra.run.smithi172.stdout:(7/9): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2023-10-05T19:46:37.667 INFO:teuthology.orchestra.run.smithi172.stdout:(8/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2023-10-05T19:46:37.684 INFO:teuthology.orchestra.run.smithi172.stdout:(9/9): libgcrypt-devel-1.8.5-4.el8.x86_64.rpm 3.0 MB/s | 149 kB 00:00 2023-10-05T19:46:37.685 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:46:37.685 INFO:teuthology.orchestra.run.smithi172.stdout:Total 10 MB/s | 1.9 MB 00:00 2023-10-05T19:46:37.693 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-10-05T19:46:37.705 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-10-05T19:46:37.705 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-10-05T19:46:37.853 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-10-05T19:46:37.857 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-10-05T19:46:38.246 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-10-05T19:46:38.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-10-05T19:46:38.470 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-10-05T19:46:38.664 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-10-05T19:46:38.678 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-10-05T19:46:38.839 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 3/9 2023-10-05T19:46:39.024 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xz-devel-5.2.4-3.el8.x86_64 4/9 2023-10-05T19:46:39.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cmake-filesystem-3.18.2-11.el8_4.x86_64 5/9 2023-10-05T19:46:39.760 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libxml2-devel-2.9.7-9.el8_4.2.x86_64 6/9 2023-10-05T19:46:39.982 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-10-05T19:46:40.133 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libffi-devel-3.1-22.el8.x86_64 8/9 2023-10-05T19:46:40.166 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libffi-devel-3.1-22.el8.x86_64 8/9 2023-10-05T19:46:40.334 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 9/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libevent-devel-2.1.8-5.el8.x86_64 9/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 1/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 2/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libxml2-devel-2.9.7-9.el8_4.2.x86_64 3/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cmake-filesystem-3.18.2-11.el8_4.x86_64 4/9 2023-10-05T19:46:40.630 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libffi-devel-3.1-22.el8.x86_64 5/9 2023-10-05T19:46:40.631 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xz-devel-5.2.4-3.el8.x86_64 6/9 2023-10-05T19:46:40.631 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/9 2023-10-05T19:46:40.631 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgcrypt-devel-1.8.5-4.el8.x86_64 8/9 2023-10-05T19:46:41.727 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 9/9 2023-10-05T19:46:41.727 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:25 ago on Thu 05 Oct 2023 07:41:16 PM UTC. 2023-10-05T19:46:43.662 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-10-05T19:46:43.762 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: cmake-filesystem-3.18.2-11.el8_4.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: libffi-devel-3.1-22.el8.x86_64 libgcrypt-devel-1.8.5-4.el8.x86_64 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-9.el8_4.2.x86_64 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-3.el8.x86_64 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:46:43.763 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-10-05T19:46:44.232 INFO:teuthology.orchestra.run.smithi172.stdout:Already using interpreter /usr/bin/python3.6 2023-10-05T19:46:44.232 INFO:teuthology.orchestra.run.smithi172.stdout:Using base prefix '/usr' 2023-10-05T19:46:44.232 INFO:teuthology.orchestra.run.smithi172.stdout:New python executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python3.6 2023-10-05T19:46:44.232 INFO:teuthology.orchestra.run.smithi172.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python 2023-10-05T19:46:47.323 INFO:teuthology.orchestra.run.smithi172.stdout:Installing setuptools, pip, wheel...done. 2023-10-05T19:46:47.664 INFO:teuthology.orchestra.run.smithi172.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-10-05T19:46:48.926 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting setuptools==44.1.0 2023-10-05T19:46:48.983 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-10-05T19:46:49.109 INFO:teuthology.orchestra.run.smithi172.stdout:Installing collected packages: setuptools 2023-10-05T19:46:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: Attempting uninstall: setuptools 2023-10-05T19:46:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: Found existing installation: setuptools 59.6.0 2023-10-05T19:46:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: Uninstalling setuptools-59.6.0: 2023-10-05T19:46:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: Successfully uninstalled setuptools-59.6.0 2023-10-05T19:46:49.395 INFO:teuthology.orchestra.run.smithi172.stdout:Successfully installed setuptools-44.1.0 2023-10-05T19:46:50.009 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting PyYAML 2023-10-05T19:46:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2023-10-05T19:46:50.226 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting nose>=1.0.0 2023-10-05T19:46:50.244 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-10-05T19:46:50.356 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting boto>=2.6.0 2023-10-05T19:46:50.377 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-10-05T19:46:51.399 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting boto3>=1.0.0 2023-10-05T19:46:51.417 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-10-05T19:46:52.532 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting botocore<1.28.0 2023-10-05T19:46:52.552 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-10-05T19:46:52.813 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting munch>=2.0.0 2023-10-05T19:46:52.831 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-10-05T19:46:53.286 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting gevent>=1.0 2023-10-05T19:46:53.307 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-10-05T19:46:53.505 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting isodate>=0.4.4 2023-10-05T19:46:53.522 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-10-05T19:46:53.631 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting requests>=2.23.0 2023-10-05T19:46:53.648 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-10-05T19:46:53.831 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting pytz>=2011k 2023-10-05T19:46:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T19:46:53.979 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting httplib2 2023-10-05T19:46:53.998 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-10-05T19:46:54.545 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting lxml 2023-10-05T19:46:54.567 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl (7.4 MB) 2023-10-05T19:46:54.834 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-10-05T19:46:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-10-05T19:46:55.167 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-10-05T19:46:55.185 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-10-05T19:46:55.269 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-10-05T19:46:55.290 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-10-05T19:46:55.421 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting urllib3<1.27,>=1.25.4 2023-10-05T19:46:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading urllib3-1.26.17-py2.py3-none-any.whl (143 kB) 2023-10-05T19:46:55.638 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting importlib-metadata>=1.7.0 2023-10-05T19:46:55.658 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2023-10-05T19:46:55.767 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting zope.event 2023-10-05T19:46:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-10-05T19:46:56.177 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting zope.interface 2023-10-05T19:46:56.197 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-10-05T19:46:56.224 INFO:teuthology.orchestra.run.smithi172.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-10-05T19:46:56.669 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting greenlet>=2.0.0 2023-10-05T19:46:56.702 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2023-10-05T19:46:56.801 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting six 2023-10-05T19:46:56.818 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T19:46:56.919 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting idna<4,>=2.5 2023-10-05T19:46:56.937 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T19:46:57.250 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting charset-normalizer~=2.0.0 2023-10-05T19:46:57.294 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-10-05T19:46:57.402 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting certifi>=2017.4.17 2023-10-05T19:46:57.419 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T19:46:57.564 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-10-05T19:46:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-10-05T19:46:57.746 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting typing-extensions>=3.6.4 2023-10-05T19:46:57.763 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2023-10-05T19:46:57.867 INFO:teuthology.orchestra.run.smithi172.stdout:Collecting zipp>=0.5 2023-10-05T19:46:57.889 INFO:teuthology.orchestra.run.smithi172.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2023-10-05T19:46:58.500 INFO:teuthology.orchestra.run.smithi172.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-10-05T19:47:02.105 INFO:teuthology.orchestra.run.smithi172.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.7.22 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.4 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.17 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2023-10-05T19:47:02.449 INFO:teuthology.orchestra.run.smithi172.stdout:running develop 2023-10-05T19:47:02.470 INFO:teuthology.orchestra.run.smithi172.stdout:running egg_info 2023-10-05T19:47:02.470 INFO:teuthology.orchestra.run.smithi172.stdout:creating s3tests.egg-info 2023-10-05T19:47:02.470 INFO:teuthology.orchestra.run.smithi172.stdout:writing s3tests.egg-info/PKG-INFO 2023-10-05T19:47:02.471 INFO:teuthology.orchestra.run.smithi172.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-10-05T19:47:02.471 INFO:teuthology.orchestra.run.smithi172.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-10-05T19:47:02.471 INFO:teuthology.orchestra.run.smithi172.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-10-05T19:47:02.471 INFO:teuthology.orchestra.run.smithi172.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:47:02.481 INFO:teuthology.orchestra.run.smithi172.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:47:02.481 INFO:teuthology.orchestra.run.smithi172.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:47:02.498 INFO:teuthology.orchestra.run.smithi172.stdout:running build_ext 2023-10-05T19:47:02.498 INFO:teuthology.orchestra.run.smithi172.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-10-05T19:47:02.498 INFO:teuthology.orchestra.run.smithi172.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-10-05T19:47:02.498 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.499 INFO:teuthology.orchestra.run.smithi172.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:47:02.499 INFO:teuthology.orchestra.run.smithi172.stdout:Processing dependencies for s3tests==0.0.1 2023-10-05T19:47:02.607 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for isodate==0.6.1 2023-10-05T19:47:02.607 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: isodate 0.6.1 2023-10-05T19:47:02.607 INFO:teuthology.orchestra.run.smithi172.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-10-05T19:47:02.608 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.608 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.609 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for gevent==22.10.2 2023-10-05T19:47:02.609 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: gevent 22.10.2 2023-10-05T19:47:02.609 INFO:teuthology.orchestra.run.smithi172.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-10-05T19:47:02.609 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for munch==4.0.0 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: munch 4.0.0 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for PyYAML==6.0.1 2023-10-05T19:47:02.610 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: PyYAML 6.0.1 2023-10-05T19:47:02.611 INFO:teuthology.orchestra.run.smithi172.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for boto3==1.23.10 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: boto3 1.23.10 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for boto==2.49.0 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: boto 2.49.0 2023-10-05T19:47:02.612 INFO:teuthology.orchestra.run.smithi172.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-10-05T19:47:02.613 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.613 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.613 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for six==1.16.0 2023-10-05T19:47:02.613 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: six 1.16.0 2023-10-05T19:47:02.613 INFO:teuthology.orchestra.run.smithi172.stdout:Adding six 1.16.0 to easy-install.pth file 2023-10-05T19:47:02.614 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.614 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.614 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for zope.interface==5.5.2 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: zope.interface 5.5.2 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for zope.event==4.6 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: zope.event 4.6 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for setuptools==44.1.0 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: setuptools 44.1.0 2023-10-05T19:47:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-10-05T19:47:02.616 INFO:teuthology.orchestra.run.smithi172.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for greenlet==2.0.2 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: greenlet 2.0.2 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.617 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.618 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for importlib-metadata==4.8.3 2023-10-05T19:47:02.618 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: importlib-metadata 4.8.3 2023-10-05T19:47:02.618 INFO:teuthology.orchestra.run.smithi172.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2023-10-05T19:47:02.618 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for s3transfer==0.5.2 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: s3transfer 0.5.2 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for jmespath==0.10.0 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: jmespath 0.10.0 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for botocore==1.26.10 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: botocore 1.26.10 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for zipp==3.6.0 2023-10-05T19:47:02.621 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: zipp 3.6.0 2023-10-05T19:47:02.622 INFO:teuthology.orchestra.run.smithi172.stdout:Adding zipp 3.6.0 to easy-install.pth file 2023-10-05T19:47:02.622 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.622 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for typing-extensions==4.1.1 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: typing-extensions 4.1.1 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.623 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for urllib3==1.26.17 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: urllib3 1.26.17 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Adding urllib3 1.26.17 to easy-install.pth file 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Searching for python-dateutil==2.8.2 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Best match: python-dateutil 2.8.2 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:47:02.624 INFO:teuthology.orchestra.run.smithi172.stdout:Finished processing dependencies for s3tests==0.0.1 2023-10-05T19:47:02.642 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:47:02.642 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-10-05T19:47:02.656 INFO:tasks.s3tests:Configuring boto... 2023-10-05T19:47:02.656 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-10-05T19:47:02.657 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T19:47:02.710 DEBUG:teuthology.orchestra.run.smithi172:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-10-05T19:47:03.199 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-10-05T19:47:03.250 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-10-05T19:47:03.322 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-10-05T19:47:03.380 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-10-05T19:47:03.439 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-10-05T19:47:03.455 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-10-05T19:47:03.513 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-10-05T19:47:03.529 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-10-05T19:47:03.570 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-10-05T19:47:03.612 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-10-05T19:47:03.665 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-10-05T19:47:03.717 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-10-05T19:47:03.769 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:47:03.821 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-10-05T19:47:03.822 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:47:03.867 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:47:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:47:03.869 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:47:03.870 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:47:04.008 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-10-05T19:47:04.009 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-10-05T19:47:04.111 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-10-05T19:47:04.159 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-10-05T19:47:06.191 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-10-05T19:47:06.286 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-10-05T19:47:06.355 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-10-05T19:47:06.379 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:47:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:47:06.468 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-10-05T19:47:06.469 INFO:teuthology.orchestra.run.smithi172.stderr:SKIP 2023-10-05T19:47:08.219 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-10-05T19:47:08.219 INFO:teuthology.orchestra.run.smithi172.stderr:/home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-10-05T19:47:08.219 INFO:teuthology.orchestra.run.smithi172.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-10-05T19:47:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-10-05T19:47:19.060 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-10-05T19:47:19.090 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-10-05T19:47:19.140 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-10-05T19:47:19.181 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-10-05T19:47:19.219 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-10-05T19:47:19.282 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-10-05T19:47:19.306 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-10-05T19:47:19.351 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-10-05T19:47:19.376 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-10-05T19:47:19.404 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-10-05T19:47:19.428 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-10-05T19:47:19.447 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-10-05T19:47:19.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-10-05T19:47:19.477 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-10-05T19:47:19.509 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-10-05T19:47:19.544 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:47:19.576 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-10-05T19:47:19.599 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:47:19.629 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-10-05T19:47:19.658 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:47:19.681 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:47:19.705 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-10-05T19:47:19.720 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:47:19.736 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-10-05T19:47:19.745 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:47:19.754 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-10-05T19:47:19.762 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:47:19.770 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-10-05T19:47:19.778 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:47:19.862 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-10-05T19:47:19.955 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-10-05T19:47:20.047 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-10-05T19:47:20.139 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-10-05T19:47:20.237 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-10-05T19:47:20.319 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-10-05T19:47:20.378 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-10-05T19:47:20.436 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-10-05T19:47:20.490 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-10-05T19:47:20.610 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-10-05T19:47:20.733 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-10-05T19:47:20.773 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:47:20.813 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:47:20.872 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-10-05T19:47:20.926 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-10-05T19:47:21.082 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-10-05T19:47:21.203 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-10-05T19:47:21.261 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-10-05T19:47:21.318 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-10-05T19:47:21.378 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-10-05T19:47:21.431 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-10-05T19:47:21.485 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-10-05T19:47:21.540 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-10-05T19:47:21.597 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-10-05T19:47:21.660 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-10-05T19:47:21.714 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-10-05T19:47:21.769 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-10-05T19:47:21.828 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-10-05T19:47:21.908 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-10-05T19:47:21.954 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-10-05T19:47:22.006 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-10-05T19:47:22.065 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-10-05T19:47:22.115 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-10-05T19:47:22.172 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-10-05T19:47:29.050 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-10-05T19:47:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-10-05T19:47:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-10-05T19:47:29.229 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-10-05T19:47:29.278 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-10-05T19:47:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-10-05T19:47:29.376 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-10-05T19:47:29.429 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-10-05T19:47:29.479 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-10-05T19:47:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-10-05T19:47:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-10-05T19:47:29.632 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-10-05T19:47:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-10-05T19:47:29.733 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-10-05T19:47:29.794 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-10-05T19:47:29.853 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-10-05T19:47:29.909 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-10-05T19:47:29.962 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:47:30.016 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:47:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:47:30.153 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:47:30.210 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:47:30.267 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:47:30.369 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-10-05T19:47:30.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-10-05T19:47:30.522 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-10-05T19:47:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-10-05T19:47:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-10-05T19:47:30.701 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-10-05T19:47:30.928 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-10-05T19:47:30.962 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-10-05T19:47:31.355 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-10-05T19:47:31.766 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-10-05T19:47:31.826 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-10-05T19:47:31.886 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-10-05T19:47:31.941 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-10-05T19:47:31.996 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-10-05T19:47:32.100 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-10-05T19:47:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-10-05T19:47:32.257 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-10-05T19:47:32.313 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-10-05T19:47:32.368 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-10-05T19:47:32.427 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-10-05T19:47:32.485 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-10-05T19:47:32.541 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-10-05T19:47:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-10-05T19:47:33.105 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-10-05T19:47:33.139 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-10-05T19:47:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-10-05T19:47:33.196 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-10-05T19:47:33.221 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-10-05T19:47:33.231 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-10-05T19:47:33.239 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-10-05T19:47:33.247 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-10-05T19:47:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-10-05T19:47:33.447 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-10-05T19:47:33.457 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-10-05T19:47:33.554 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-10-05T19:47:33.578 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-10-05T19:47:33.605 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-10-05T19:47:33.829 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-10-05T19:47:34.054 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-10-05T19:47:41.090 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-10-05T19:47:48.279 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-10-05T19:47:48.314 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-10-05T19:47:48.342 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-10-05T19:47:48.375 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-10-05T19:47:48.413 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-10-05T19:47:48.544 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-10-05T19:47:48.580 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-10-05T19:47:48.618 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-10-05T19:47:48.675 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-10-05T19:47:48.738 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-10-05T19:47:48.811 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-10-05T19:47:48.884 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-10-05T19:47:48.919 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-10-05T19:47:48.985 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-10-05T19:47:49.005 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-10-05T19:47:49.030 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-10-05T19:47:49.057 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-10-05T19:47:49.114 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-10-05T19:47:49.147 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-10-05T19:47:49.175 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-10-05T19:47:49.203 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-10-05T19:47:49.239 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-10-05T19:47:49.313 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-10-05T19:47:49.338 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-10-05T19:47:49.361 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-10-05T19:47:49.384 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-10-05T19:47:49.409 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-10-05T19:47:49.433 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-10-05T19:47:49.459 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-10-05T19:47:49.492 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-10-05T19:47:49.515 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-10-05T19:47:49.538 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-10-05T19:47:49.563 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-10-05T19:47:49.591 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-10-05T19:47:49.617 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-10-05T19:47:49.642 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-10-05T19:47:49.664 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-10-05T19:47:49.689 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-10-05T19:47:49.713 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-10-05T19:47:49.740 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-10-05T19:47:49.767 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-10-05T19:47:49.843 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-10-05T19:47:49.871 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-10-05T19:47:49.945 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-10-05T19:47:49.979 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-10-05T19:47:50.082 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-10-05T19:47:50.128 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-10-05T19:47:51.171 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-10-05T19:47:51.248 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-10-05T19:47:51.326 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-10-05T19:47:51.451 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-10-05T19:47:51.615 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-10-05T19:47:51.764 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-10-05T19:47:51.876 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-10-05T19:47:52.000 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-10-05T19:47:52.161 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-10-05T19:47:52.236 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-10-05T19:47:52.398 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-10-05T19:47:52.429 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-10-05T19:47:52.501 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-10-05T19:47:52.579 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-10-05T19:47:52.624 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-10-05T19:47:52.650 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-10-05T19:47:52.658 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-10-05T19:47:52.719 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-10-05T19:47:52.754 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-10-05T19:47:52.793 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-10-05T19:47:52.825 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-10-05T19:47:52.858 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-10-05T19:47:52.890 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-10-05T19:47:52.927 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-10-05T19:47:53.040 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-10-05T19:47:53.124 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-10-05T19:47:53.159 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-10-05T19:47:53.191 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-10-05T19:47:53.221 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-10-05T19:47:53.282 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-10-05T19:47:53.319 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-10-05T19:47:53.362 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-10-05T19:47:53.395 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-10-05T19:47:53.426 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-10-05T19:47:53.437 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-10-05T19:47:53.447 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-10-05T19:47:53.459 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-10-05T19:47:53.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-10-05T19:47:53.494 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-10-05T19:47:53.557 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-10-05T19:47:53.587 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-10-05T19:47:53.653 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-10-05T19:47:53.717 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-10-05T19:47:53.741 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-10-05T19:47:53.804 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-10-05T19:47:53.915 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-10-05T19:47:53.977 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-10-05T19:47:54.040 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-10-05T19:47:54.113 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-10-05T19:47:54.192 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-10-05T19:47:54.317 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-10-05T19:47:54.396 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-10-05T19:47:54.471 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-10-05T19:47:54.596 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-10-05T19:47:54.721 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-10-05T19:47:54.809 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-10-05T19:47:54.863 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-10-05T19:47:54.892 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-10-05T19:47:55.017 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-10-05T19:47:55.139 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-10-05T19:47:55.264 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-10-05T19:47:55.415 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-10-05T19:47:55.541 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-10-05T19:47:55.661 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-10-05T19:47:55.773 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-10-05T19:47:55.890 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-10-05T19:47:56.006 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-10-05T19:47:56.121 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-10-05T19:47:56.155 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-10-05T19:47:56.260 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-10-05T19:47:56.334 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-10-05T19:47:56.415 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-10-05T19:47:56.499 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-10-05T19:47:56.533 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-10-05T19:47:56.660 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-10-05T19:47:56.891 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-10-05T19:47:57.101 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-10-05T19:47:57.308 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-10-05T19:47:57.519 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-10-05T19:47:57.722 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-10-05T19:47:57.926 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-10-05T19:47:58.142 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-10-05T19:47:58.351 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-10-05T19:47:58.592 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-10-05T19:47:58.760 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-10-05T19:47:58.931 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-10-05T19:47:59.099 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-10-05T19:47:59.283 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-10-05T19:47:59.423 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-10-05T19:47:59.429 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-10-05T19:47:59.437 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-10-05T19:47:59.445 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-10-05T19:47:59.611 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-10-05T19:47:59.778 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-10-05T19:47:59.795 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-10-05T19:47:59.812 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-10-05T19:47:59.876 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-10-05T19:48:00.549 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-10-05T19:48:00.629 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-10-05T19:48:00.727 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-10-05T19:48:00.878 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-10-05T19:48:01.006 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-10-05T19:48:01.094 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-10-05T19:48:01.181 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-10-05T19:48:01.327 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-10-05T19:48:01.380 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-10-05T19:48:01.496 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-10-05T19:48:01.637 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-10-05T19:48:01.859 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-10-05T19:48:02.046 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-10-05T19:48:02.082 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-10-05T19:48:02.115 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-10-05T19:48:02.688 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-10-05T19:48:02.842 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-10-05T19:48:03.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-10-05T19:48:03.942 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-10-05T19:48:04.004 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-10-05T19:48:04.281 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-10-05T19:48:04.368 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-10-05T19:48:04.603 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-10-05T19:48:05.559 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-10-05T19:48:06.332 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-10-05T19:48:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-10-05T19:48:12.152 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-10-05T19:48:12.992 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-10-05T19:48:14.396 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-10-05T19:48:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-10-05T19:48:14.909 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-10-05T19:48:15.254 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-10-05T19:48:15.436 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-10-05T19:48:15.513 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-10-05T19:48:15.821 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-10-05T19:48:15.969 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-10-05T19:48:16.060 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-10-05T19:48:16.146 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-10-05T19:48:16.171 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-10-05T19:48:16.298 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-10-05T19:48:19.420 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-10-05T19:48:22.467 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-10-05T19:48:25.513 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-10-05T19:48:25.647 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-10-05T19:48:25.751 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-10-05T19:48:25.938 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-10-05T19:48:26.237 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-10-05T19:48:26.410 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-10-05T19:48:26.649 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-10-05T19:48:27.042 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-10-05T19:48:27.167 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-10-05T19:48:27.432 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-10-05T19:48:27.888 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-10-05T19:48:28.015 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-10-05T19:48:28.185 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-10-05T19:48:28.454 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-10-05T19:48:28.531 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-10-05T19:48:34.332 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-10-05T19:48:34.406 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-10-05T19:48:34.507 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-10-05T19:48:34.582 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-10-05T19:48:34.660 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-10-05T19:48:34.900 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-10-05T19:48:40.795 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-10-05T19:48:41.103 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-10-05T19:48:41.305 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-10-05T19:48:41.612 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-10-05T19:48:41.960 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-10-05T19:48:44.342 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-10-05T19:48:47.494 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-10-05T19:48:59.918 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-10-05T19:49:01.414 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-10-05T19:49:02.110 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-10-05T19:49:02.894 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-10-05T19:49:03.170 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-10-05T19:49:03.469 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-10-05T19:49:03.602 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-10-05T19:49:03.873 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-10-05T19:49:04.052 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-10-05T19:49:04.522 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-10-05T19:49:04.558 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-10-05T19:49:04.633 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-10-05T19:49:04.708 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-10-05T19:49:04.744 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-10-05T19:49:04.776 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-10-05T19:49:04.898 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-10-05T19:49:04.934 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-10-05T19:49:04.962 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-10-05T19:49:05.022 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-10-05T19:49:05.058 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-10-05T19:49:05.090 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-10-05T19:49:05.122 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-10-05T19:49:05.160 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-10-05T19:49:05.197 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-10-05T19:49:05.198 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-10-05T19:49:05.279 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-10-05T19:49:05.356 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-10-05T19:49:05.417 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-10-05T19:49:05.496 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-10-05T19:49:05.540 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-10-05T19:49:05.619 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-10-05T19:49:05.697 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-10-05T19:49:05.796 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-10-05T19:49:05.871 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-10-05T19:49:05.941 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-10-05T19:49:06.010 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-10-05T19:49:06.917 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-10-05T19:49:07.514 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-10-05T19:49:07.555 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-10-05T19:49:07.607 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-10-05T19:49:07.685 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-10-05T19:49:07.753 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-10-05T19:49:07.783 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-10-05T19:49:08.695 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-10-05T19:49:09.217 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-10-05T19:49:09.736 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-10-05T19:49:09.781 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-10-05T19:49:09.863 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-10-05T19:49:09.944 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-10-05T19:49:10.001 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-10-05T19:49:10.078 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-10-05T19:49:10.159 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-10-05T19:49:10.206 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-10-05T19:49:10.260 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-10-05T19:49:10.349 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-10-05T19:49:10.409 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-10-05T19:49:10.548 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-10-05T19:49:10.682 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-10-05T19:49:10.861 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-10-05T19:49:10.989 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-10-05T19:49:11.169 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-10-05T19:49:11.343 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-10-05T19:49:11.517 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-10-05T19:49:11.694 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-10-05T19:49:11.868 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-10-05T19:49:12.091 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-10-05T19:49:12.307 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-10-05T19:49:12.427 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-10-05T19:49:12.466 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-10-05T19:49:12.589 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-10-05T19:49:12.735 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-10-05T19:49:12.924 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-10-05T19:49:13.118 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-10-05T19:49:13.381 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-10-05T19:49:15.120 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-10-05T19:49:15.292 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-10-05T19:49:15.517 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-10-05T19:49:15.701 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-10-05T19:49:15.983 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-10-05T19:49:16.169 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-10-05T19:49:16.260 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-10-05T19:49:16.432 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-10-05T19:49:16.462 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-10-05T19:49:16.501 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-10-05T19:49:16.536 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-10-05T19:49:16.568 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-10-05T19:49:16.654 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-10-05T19:49:16.727 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-10-05T19:49:16.957 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-10-05T19:49:17.031 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-10-05T19:49:17.102 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-10-05T19:49:17.208 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-10-05T19:49:17.278 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-10-05T19:49:17.345 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-10-05T19:49:17.457 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-10-05T19:49:17.530 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-10-05T19:49:17.599 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-10-05T19:49:17.671 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-10-05T19:49:17.739 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-10-05T19:49:17.814 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-10-05T19:49:17.890 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-10-05T19:49:18.018 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-10-05T19:49:18.135 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-10-05T19:49:18.246 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-10-05T19:49:18.343 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-10-05T19:49:18.495 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-10-05T19:49:18.646 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-10-05T19:49:18.794 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-10-05T19:49:18.932 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-10-05T19:49:19.079 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-10-05T19:49:19.218 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-10-05T19:49:19.385 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-10-05T19:49:19.533 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-10-05T19:49:19.592 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-10-05T19:49:19.664 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-10-05T19:49:19.744 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-10-05T19:49:19.886 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-10-05T19:49:19.968 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-10-05T19:49:20.073 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-10-05T19:49:20.126 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-10-05T19:49:20.207 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-10-05T19:49:20.274 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-10-05T19:49:20.314 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-10-05T19:49:20.395 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-10-05T19:49:20.482 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-10-05T19:49:20.613 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-10-05T19:49:20.746 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-10-05T19:49:20.777 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-10-05T19:49:20.856 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-10-05T19:49:20.932 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-10-05T19:49:21.014 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-10-05T19:49:21.097 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-10-05T19:49:21.179 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-10-05T19:49:21.241 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-10-05T19:49:21.322 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-10-05T19:49:21.557 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-10-05T19:49:21.720 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-10-05T19:49:21.795 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-10-05T19:49:21.984 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-10-05T19:49:22.838 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-10-05T19:49:22.876 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-10-05T19:49:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-10-05T19:49:23.053 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-10-05T19:49:23.139 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-10-05T19:49:24.083 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-10-05T19:49:25.023 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-10-05T19:49:25.080 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-10-05T19:49:25.491 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-10-05T19:49:26.051 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-10-05T19:49:26.324 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-10-05T19:49:26.530 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-10-05T19:49:26.577 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-10-05T19:49:27.125 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-10-05T19:49:27.485 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-10-05T19:49:27.998 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-10-05T19:49:28.313 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-10-05T19:49:28.587 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-10-05T19:49:28.774 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-10-05T19:49:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-10-05T19:49:29.140 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-10-05T19:49:29.407 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-10-05T19:49:29.497 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-10-05T19:49:29.712 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-10-05T19:49:29.925 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-10-05T19:49:30.155 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-10-05T19:49:30.412 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-10-05T19:49:30.624 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-10-05T19:49:30.624 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-10-05T19:49:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-10-05T19:49:31.366 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-10-05T19:49:31.617 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-10-05T19:49:31.813 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-10-05T19:49:31.996 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-10-05T19:49:45.007 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-10-05T19:49:45.007 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-10-05T19:50:44.257 INFO:teuthology.orchestra.run.smithi172.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-10-05T19:50:44.258 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:50:44.258 INFO:teuthology.orchestra.run.smithi172.stderr:---------------------------------------------------------------------- 2023-10-05T19:50:44.258 INFO:teuthology.orchestra.run.smithi172.stderr:Ran 618 tests in 221.335s 2023-10-05T19:50:44.258 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:50:44.258 INFO:teuthology.orchestra.run.smithi172.stderr:OK (SKIP=79) 2023-10-05T19:50:44.322 INFO:teuthology.orchestra.run.smithi172.stderr:Exception ignored in: .generate at 0x7f4f9c448620> 2023-10-05T19:50:44.322 INFO:teuthology.orchestra.run.smithi172.stderr:RuntimeError: generator ignored GeneratorExit 2023-10-05T19:50:44.350 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-10-05T19:50:44.350 DEBUG:teuthology.orchestra.run.smithi172:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-10-05T19:50:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:50:44.418 INFO:tasks.s3tests:Cleaning up boto... 2023-10-05T19:50:44.418 DEBUG:teuthology.orchestra.run.smithi172:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T19:50:44.432 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-10-05T19:50:44.513 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:50:44.513 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:50:44.519 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.517+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 20 realm 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.518+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.519+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.521 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.519+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.519+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.519+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.520+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:44.522 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.520+0000 7f86af457680 20 searching for the correct realm 2023-10-05T19:50:44.529 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:44.529 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.530 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.527+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:44.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.531 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.528+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.532 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.529+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:44.537 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:44.538 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.535+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 zone default found 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 4 Realm: () 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:50:44.540 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 zonegroup default 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.536+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.537+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.537+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.537+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.537+0000 7f86af457680 20 started sync module instance, tier type = 2023-10-05T19:50:44.541 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.537+0000 7f86af457680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:50:44.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=0 2023-10-05T19:50:44.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=1 2023-10-05T19:50:44.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=2 2023-10-05T19:50:44.551 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=3 2023-10-05T19:50:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=4 2023-10-05T19:50:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=5 2023-10-05T19:50:44.552 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.549+0000 7f86af457680 20 add_watcher() i=6 2023-10-05T19:50:44.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.552+0000 7f86af457680 20 add_watcher() i=7 2023-10-05T19:50:44.554 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.552+0000 7f86af457680 2 all 8 watchers are set, enabling cache 2023-10-05T19:50:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.553+0000 7f8673fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.553+0000 7f86af457680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:50:44.555 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.553+0000 7f86af457680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f85e2ffd700 20 reqs_thread_entry: start 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f85e0ff9700 20 reqs_thread_entry: start 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f86af457680 20 init_complete bucket index max shards: 11 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f86af457680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f85d6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:51:14 2023 2023-10-05T19:50:44.606 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f85d67fc700 20 reqs_thread_entry: start 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f86af457680 20 get_system_obj_state: rctx=0x7fff2850daf8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.603+0000 7f86af457680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 20 Read xattr: user.rgw.idtag 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T19:50:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T19:50:44.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:50:44.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.604+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 get_system_obj_state: rctx=0x7fff2850d7b0 obj=default.rgw.meta:root:.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 : miss 2023-10-05T19:50:44.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 info.flags=0x16 2023-10-05T19:50:44.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 adding default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 to cache LRU end 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 Read xattr: user.rgw.acl 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 Read xattr: user.rgw.idtag 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=364 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 info.flags=0x13 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 moving default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 to cache LRU end 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.605+0000 7f86af457680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.606+0000 7f86af457680 10 cls_bucket_list_unordered: got 1236A3 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.606+0000 7f86af457680 10 cls_bucket_list_unordered: got A4A988 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.606+0000 7f86af457680 10 cls_bucket_list_unordered: got D15589 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.606+0000 7f86af457680 10 cls_bucket_list_unordered: got 184194 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.606+0000 7f86af457680 10 cls_bucket_list_unordered: got D5FF63 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.607+0000 7f86af457680 10 cls_bucket_list_unordered: got 47C10D 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.607+0000 7f86af457680 10 cls_bucket_list_unordered: got 51209C 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.607+0000 7f86af457680 10 cls_bucket_list_unordered: got E9A028 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.607+0000 7f86af457680 10 cls_bucket_list_unordered: got 68B807 2023-10-05T19:50:44.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.607+0000 7f86af457680 10 cls_bucket_list_unordered: got 86C0D5 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got C8C4C8 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got D55E6D 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got E067AC 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got 05E7F3 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got 16628A 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got 772F4C 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got 026337 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got DF7032 2023-10-05T19:50:44.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got E3A198 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.608+0000 7f86af457680 10 cls_bucket_list_unordered: got F162F9 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got 6FF1EF 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got D795EF 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got FB60F9 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got 31CD9B 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got 8E956B 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got C93639 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got F0BD2C 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got FCFBE4 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 10 cls_bucket_list_unordered: got EAC243 2023-10-05T19:50:44.614 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.609+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:1236A3 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.610+0000 7f86af457680 10 manifest: total_size = 360872 2023-10-05T19:50:44.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.610+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.7201013672203869839 2023-10-05T19:50:44.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.610+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:1236A3 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.610+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.610+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.0 2023-10-05T19:50:44.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-10-05T19:50:44.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.616 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.612+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:A4A988 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f86af457680 10 manifest: total_size = 399130 2023-10-05T19:50:44.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.4660776721283299235 2023-10-05T19:50:44.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:A4A988 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.0 2023-10-05T19:50:44.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.613+0000 7f8693fff700 20 handle_completion(): completion ok for obj=1236A3 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D15589 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 10 manifest: total_size = 589748 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.10640608937160309387 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D15589 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.619 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.615+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.0 2023-10-05T19:50:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.616+0000 7f86937fe700 20 handle_completion(): completion ok for obj=A4A988 2023-10-05T19:50:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:184194 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 10 manifest: total_size = 399129 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.1484506432869547717 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:184194 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.618+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.1 2023-10-05T19:50:44.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.619+0000 7f8693fff700 20 handle_completion(): completion ok for obj=D15589 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.620+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D5FF63 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.621+0000 7f86af457680 10 manifest: total_size = 399252 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.621+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.17354350616167741580 2023-10-05T19:50:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.621+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D5FF63 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.624 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.621+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.624 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.621+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.1 2023-10-05T19:50:44.624 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.622+0000 7f86937fe700 20 handle_completion(): completion ok for obj=184194 2023-10-05T19:50:44.628 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:47C10D state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 10 manifest: total_size = 399272 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.17323452746870672771 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:47C10D state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.629 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.626+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.2 2023-10-05T19:50:44.630 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.629+0000 7f8693fff700 20 handle_completion(): completion ok for obj=D5FF63 2023-10-05T19:50:44.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.636 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:51209C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 10 manifest: total_size = 399169 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.13757444217048020204 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:51209C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.637 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.634+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.2 2023-10-05T19:50:44.638 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.635+0000 7f86937fe700 20 handle_completion(): completion ok for obj=47C10D 2023-10-05T19:50:44.641 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.641 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.641 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.641 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E9A028 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 10 manifest: total_size = 399397 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.17005964604829354889 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E9A028 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.642 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.639+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.3 2023-10-05T19:50:44.645 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.643+0000 7f8693fff700 20 handle_completion(): completion ok for obj=51209C 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.644+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:68B807 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.645+0000 7f86af457680 10 manifest: total_size = 5 2023-10-05T19:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.645+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.17461292971170527745 2023-10-05T19:50:44.648 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.645+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:68B807 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.648 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.645+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.648 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.645+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.4 2023-10-05T19:50:44.648 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.646+0000 7f86937fe700 20 handle_completion(): completion ok for obj=E9A028 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:86C0D5 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.654 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 10 manifest: total_size = 4 2023-10-05T19:50:44.655 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.3259665696088030718 2023-10-05T19:50:44.655 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:86C0D5 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.655 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.655 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.652+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.4 2023-10-05T19:50:44.656 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.654+0000 7f8693fff700 20 handle_completion(): completion ok for obj=68B807 2023-10-05T19:50:44.661 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.662 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.662 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.662 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.662 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.662 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.659+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:C8C4C8 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.663 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.660+0000 7f86af457680 10 manifest: total_size = 180000 2023-10-05T19:50:44.663 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.660+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.7294943677710986416 2023-10-05T19:50:44.663 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.660+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:C8C4C8 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.663 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.660+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.663 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.660+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.5 2023-10-05T19:50:44.664 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.662+0000 7f86937fe700 20 handle_completion(): completion ok for obj=86C0D5 2023-10-05T19:50:44.669 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D55E6D state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 10 manifest: total_size = 399138 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.6417801715427894313 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D55E6D state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.670 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.667+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.5 2023-10-05T19:50:44.671 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.668+0000 7f8693fff700 20 handle_completion(): completion ok for obj=C8C4C8 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.673 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E067AC state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 10 manifest: total_size = 49213 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.9786096868644694269 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E067AC state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.671+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.5 2023-10-05T19:50:44.674 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.672+0000 7f86937fe700 20 handle_completion(): completion ok for obj=D55E6D 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.677+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:05E7F3 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.678+0000 7f86af457680 10 manifest: total_size = 41 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.678+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.16129575131000028504 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.678+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:05E7F3 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.678+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.681 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.678+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.6 2023-10-05T19:50:44.682 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.680+0000 7f8693fff700 20 handle_completion(): completion ok for obj=E067AC 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:16628A state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 10 manifest: total_size = 399088 2023-10-05T19:50:44.688 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.2610076292049470153 2023-10-05T19:50:44.689 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:16628A state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.689 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.689 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.686+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.6 2023-10-05T19:50:44.690 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.688+0000 7f86937fe700 20 handle_completion(): completion ok for obj=05E7F3 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.692+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:772F4C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.693+0000 7f86af457680 10 manifest: total_size = 643200 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.693+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.10324340263988797044 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.693+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:772F4C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.693+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.695 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.693+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.6 2023-10-05T19:50:44.696 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.694+0000 7f8693fff700 20 handle_completion(): completion ok for obj=16628A 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:026337 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 10 manifest: total_size = 399323 2023-10-05T19:50:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.5378045531970002031 2023-10-05T19:50:44.701 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:026337 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.701 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.701 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.698+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.7 2023-10-05T19:50:44.701 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.699+0000 7f86937fe700 20 handle_completion(): completion ok for obj=772F4C 2023-10-05T19:50:44.704 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.704 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.704 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:DF7032 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 10 manifest: total_size = 399187 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.15005346534030619855 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:DF7032 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.702+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.7 2023-10-05T19:50:44.705 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.703+0000 7f8693fff700 20 handle_completion(): completion ok for obj=026337 2023-10-05T19:50:44.709 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.709 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.709 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.709 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E3A198 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 10 manifest: total_size = 117 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.14933206979396372122 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:E3A198 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.707+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.7 2023-10-05T19:50:44.710 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.708+0000 7f86937fe700 20 handle_completion(): completion ok for obj=DF7032 2023-10-05T19:50:44.714 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.714 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:F162F9 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 10 manifest: total_size = 39912460 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.12899899577242520800 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:F162F9 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.715 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.712+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.7 2023-10-05T19:50:44.716 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.713+0000 7f8693fff700 20 handle_completion(): completion ok for obj=E3A198 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 RGWObjManifest::operator++(): result: ofs=39912460 stripe_ofs=39912460 part_ofs=0 rule->part_size=0 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - tag is: 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.12899899577242520800 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_1 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 333 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_2 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 565 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_3 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 797 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_4 2023-10-05T19:50:44.720 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1029 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_5 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1261 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_6 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1493 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_7 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1725 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_8 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1957 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: 5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518__shadow_.f02Y5nG7jER6EYn7tkllC09uRiV_4NB_9 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 2189 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-10-05T19:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.715+0000 7f86af457680 20 garbage collection: RGWGC::send_chain - on object name: gc.15tag is: 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.12899899577242520800 2023-10-05T19:50:44.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.716+0000 7f86937fe700 20 handle_completion(): completion ok for obj=F162F9 2023-10-05T19:50:44.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:6FF1EF state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 10 manifest: total_size = 399064 2023-10-05T19:50:44.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.17039438486819983378 2023-10-05T19:50:44.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:6FF1EF state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.717+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.8 2023-10-05T19:50:44.724 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D795EF state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.722+0000 7f86af457680 10 manifest: total_size = 576104 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.723+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.16289300014870141785 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.723+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:D795EF state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.723+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.725 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.723+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.8 2023-10-05T19:50:44.727 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.725+0000 7f8693fff700 20 handle_completion(): completion ok for obj=6FF1EF 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.727+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:FB60F9 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.728+0000 7f86af457680 10 manifest: total_size = 399167 2023-10-05T19:50:44.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.728+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.16215913877025352129 2023-10-05T19:50:44.731 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.728+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:FB60F9 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.731 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.728+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.731 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.728+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.8 2023-10-05T19:50:44.731 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.729+0000 7f86937fe700 20 handle_completion(): completion ok for obj=D795EF 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:31CD9B state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 10 manifest: total_size = 399109 2023-10-05T19:50:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.1877846273058382174 2023-10-05T19:50:44.734 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:31CD9B state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.734 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.734 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.731+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.9 2023-10-05T19:50:44.734 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.732+0000 7f8693fff700 20 handle_completion(): completion ok for obj=FB60F9 2023-10-05T19:50:44.735 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.733+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:8E956B state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86af457680 10 manifest: total_size = 643768 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.2112080452542774506 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:8E956B state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.736 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.9 2023-10-05T19:50:44.737 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.734+0000 7f86937fe700 20 handle_completion(): completion ok for obj=31CD9B 2023-10-05T19:50:44.738 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:C93639 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 10 manifest: total_size = 573942 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.4065596865470358017 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:C93639 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.739 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.736+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.9 2023-10-05T19:50:44.740 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.737+0000 7f8693fff700 20 handle_completion(): completion ok for obj=8E956B 2023-10-05T19:50:44.741 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.741 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.741 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.741 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:F0BD2C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 10 manifest: total_size = 398891 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.13890205589891224953 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:F0BD2C state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.739+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.9 2023-10-05T19:50:44.742 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.740+0000 7f86937fe700 20 handle_completion(): completion ok for obj=C93639 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.744 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:FCFBE4 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 10 manifest: total_size = 399250 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.3806390990219350912 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:FCFBE4 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.742+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.9 2023-10-05T19:50:44.745 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.743+0000 7f8693fff700 20 handle_completion(): completion ok for obj=F0BD2C 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.747 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:EAC243 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 10 manifest: total_size = 180000 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.2231144169840715600 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=test:EAC243 state=0x557d0bda9758 s->prefetch_data=0 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda9758 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.745+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518.10 2023-10-05T19:50:44.748 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.746+0000 7f86937fe700 20 handle_completion(): completion ok for obj=FCFBE4 2023-10-05T19:50:44.750 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:50:44.751 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.748+0000 7f86af457680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.750+0000 7f8693fff700 20 handle_completion(): completion ok for obj=EAC243 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518])"; uploads.size()=0; is_truncated=0 2023-10-05T19:50:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]) 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 lifecycle: get_system_obj_state: rctx=0x7fff2850d288 obj=default.rgw.meta:root:test state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:44.754 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:50:44.755 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.751+0000 7f86af457680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.752+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 info.flags=0x17 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.752+0000 7f86af457680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 to cache LRU end 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.752+0000 7f86af457680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.752+0000 7f86af457680 10 lifecycle: distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518, ofs0, ns] 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888147 cookie 93995557041696 notifier 4246 bl.length()=800 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f8693fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 info.flags=0x17 2023-10-05T19:50:44.756 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f8693fff700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 to cache LRU end 2023-10-05T19:50:44.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f8693fff700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]): orig_sources= new_sources= 2023-10-05T19:50:44.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]): potential sources added=[] removed=[] 2023-10-05T19:50:44.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]): orig_dests= new_dests= 2023-10-05T19:50:44.757 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.753+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:test[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518]): potential dests added=[] removed=[] 2023-10-05T19:50:44.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.762+0000 7f86af457680 20 get_system_obj_state: rctx=0x7fff2850cce8 obj=default.rgw.meta:root:test state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.762+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 moving default.rgw.meta+root+test to cache LRU end 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 removing default.rgw.meta+root+test from cache 2023-10-05T19:50:44.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-10-05T19:50:44.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.763+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889643 cookie 93995557059472 notifier 4246 bl.length()=148 2023-10-05T19:50:44.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.764+0000 7f86af457680 10 removing default.rgw.meta+root+.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 from cache 2023-10-05T19:50:44.766 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.764+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518, ofs0, ns] 2023-10-05T19:50:44.767 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.765+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888148 cookie 93995557041696 notifier 4246 bl.length()=207 2023-10-05T19:50:44.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.772+0000 7f86af457680 10 removing default.rgw.log++pubsub..bucket.test/5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518 from cache 2023-10-05T19:50:44.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.772+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/5cb8517b-2313-4992-9a76-073dfd099cc7.4206.518, ofs0, ns] 2023-10-05T19:50:44.774 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.772+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889644 cookie 93995557039488 notifier 4246 bl.length()=204 2023-10-05T19:50:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 20 get_system_obj_state: rctx=0x7fff2850d7b0 obj=default.rgw.meta:root:.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.778 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 : miss 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 info.flags=0x16 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 to cache LRU end 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 20 Read xattr: user.rgw.acl 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 20 Read xattr: user.rgw.idtag 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:44.779 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.776+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 info.flags=0x13 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 to cache LRU end 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 2023-10-05T19:50:44.780 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.777+0000 7f86af457680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.780+0000 7f86af457680 10 cls_bucket_list_unordered: got 0B6EA8 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.780+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=testbuck2:0B6EA8 state=0x557d0bda6838 s->prefetch_data=0 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.781+0000 7f86af457680 10 manifest: total_size = 399187 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.781+0000 7f86af457680 20 get_obj_state: setting s->obj_tag to 5cb8517b-2313-4992-9a76-073dfd099cc7.4204.11600899698903143630 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.781+0000 7f86af457680 20 get_obj_state: rctx=0x7fff2850d7c0 obj=testbuck2:0B6EA8 state=0x557d0bda6838 s->prefetch_data=0 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.781+0000 7f86af457680 20 prepare_atomic_modification: state is not atomic. state=0x557d0bda6838 2023-10-05T19:50:44.783 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.781+0000 7f86af457680 20 bucket index object: default.rgw.buckets.index:.dir.5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519.10 2023-10-05T19:50:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-10-05T19:50:44.787 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 chain_cache_entry: cache_locator= 2023-10-05T19:50:44.788 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:50:44.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:50:44.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:50:44.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:50:44.789 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.785+0000 7f86af457680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86937fe700 20 handle_completion(): completion ok for obj=0B6EA8 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519])"; uploads.size()=0; is_truncated=0 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]) 2023-10-05T19:50:44.790 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 20 lifecycle: get_system_obj_state: rctx=0x7fff2850d288 obj=default.rgw.meta:root:testbuck2 state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.787+0000 7f86af457680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:50:44.791 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.788+0000 7f86af457680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.792 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 info.flags=0x17 2023-10-05T19:50:44.792 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 to cache LRU end 2023-10-05T19:50:44.792 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.792 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 10 lifecycle: distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519, ofs0, ns] 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888149 cookie 93995557041696 notifier 4246 bl.length()=810 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86937fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 info.flags=0x17 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86937fe700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 to cache LRU end 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86937fe700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]): orig_sources= new_sources= 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]): potential sources added=[] removed=[] 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]): orig_dests= new_dests= 2023-10-05T19:50:44.793 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.790+0000 7f86af457680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519]): potential dests added=[] removed=[] 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 20 get_system_obj_state: rctx=0x7fff2850cce8 obj=default.rgw.meta:root:testbuck2 state=0x557d0bb11240 s->prefetch_data=0 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:44.802 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:44.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-10-05T19:50:44.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-10-05T19:50:44.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-10-05T19:50:44.803 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.799+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889645 cookie 93995557051584 notifier 4246 bl.length()=153 2023-10-05T19:50:44.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.802+0000 7f86af457680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 from cache 2023-10-05T19:50:44.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.802+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519, ofs0, ns] 2023-10-05T19:50:44.804 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.802+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888150 cookie 93995557041696 notifier 4246 bl.length()=212 2023-10-05T19:50:44.815 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.813+0000 7f86af457680 10 removing default.rgw.log++pubsub..bucket.testbuck2/5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519 from cache 2023-10-05T19:50:44.815 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.813+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/5cb8517b-2313-4992-9a76-073dfd099cc7.4206.519, ofs0, ns] 2023-10-05T19:50:44.815 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.813+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889646 cookie 93995557039488 notifier 4246 bl.length()=209 2023-10-05T19:50:44.818 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.816+0000 7f86af457680 10 removing key index: CYBQAIVZPJQSUWJJEDQA 2023-10-05T19:50:44.818 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.816+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:CYBQAIVZPJQSUWJJEDQA, ofs0, ns] 2023-10-05T19:50:44.818 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.816+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889647 cookie 93995557039488 notifier 4246 bl.length()=170 2023-10-05T19:50:44.821 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.819+0000 7f86af457680 10 removing email index: foo.client.0+test@test.test 2023-10-05T19:50:44.821 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.819+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T19:50:44.821 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.819+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889648 cookie 93995557059472 notifier 4246 bl.length()=178 2023-10-05T19:50:44.822 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.820+0000 7f86af457680 10 removing user buckets index 2023-10-05T19:50:44.823 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.820+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-10-05T19:50:44.823 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.821+0000 7f8693fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889649 cookie 93995557039488 notifier 4246 bl.length()=169 2023-10-05T19:50:44.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.823+0000 7f86af457680 10 removing user index: foo.client.0 2023-10-05T19:50:44.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.823+0000 7f86af457680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-10-05T19:50:44.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.823+0000 7f86af457680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:50:44.825 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.823+0000 7f86937fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889696 cookie 93995557038384 notifier 4246 bl.length()=161 2023-10-05T19:50:44.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.825+0000 7f86af457680 20 remove_watcher() i=0 2023-10-05T19:50:44.827 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.825+0000 7f86af457680 2 removed watcher, disabling cache 2023-10-05T19:50:44.828 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.826+0000 7f86af457680 20 remove_watcher() i=1 2023-10-05T19:50:44.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.827+0000 7f86af457680 20 remove_watcher() i=2 2023-10-05T19:50:44.829 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.828+0000 7f86af457680 20 remove_watcher() i=3 2023-10-05T19:50:44.830 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.828+0000 7f86af457680 20 remove_watcher() i=4 2023-10-05T19:50:44.831 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.829+0000 7f86af457680 20 remove_watcher() i=5 2023-10-05T19:50:44.834 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.832+0000 7f86af457680 20 remove_watcher() i=6 2023-10-05T19:50:44.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.833+0000 7f86af457680 20 remove_watcher() i=7 2023-10-05T19:50:44.835 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.834+0000 7f8673fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:44.841 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-10-05T19:50:44.920 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:50:44.920 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:50:44.926 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.924+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.924+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 realm 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.927 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.925+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.926+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.926+0000 7f593b7f7680 20 searching for the correct realm 2023-10-05T19:50:44.935 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.932+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.932+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.932+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.932+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.936 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.937 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.937 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.933+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.937 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.934+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.943 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.940+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 zone default found 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 4 Realm: () 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:50:44.944 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 zonegroup default 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.945 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:44.946 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:44.946 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 started sync module instance, tier type = 2023-10-05T19:50:44.946 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.941+0000 7f593b7f7680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=0 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=1 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=2 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=3 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=4 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=5 2023-10-05T19:50:44.959 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.957+0000 7f593b7f7680 20 add_watcher() i=6 2023-10-05T19:50:44.960 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.958+0000 7f593b7f7680 20 add_watcher() i=7 2023-10-05T19:50:44.960 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.958+0000 7f593b7f7680 2 all 8 watchers are set, enabling cache 2023-10-05T19:50:44.961 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.959+0000 7f58fffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:44.961 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.959+0000 7f593b7f7680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:50:44.961 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:44.959+0000 7f593b7f7680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:50:45.033 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f586effd700 20 reqs_thread_entry: start 2023-10-05T19:50:45.033 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f586cff9700 20 reqs_thread_entry: start 2023-10-05T19:50:45.033 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 20 init_complete bucket index max shards: 11 2023-10-05T19:50:45.033 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f5862ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:51:15 2023 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f58627fc700 20 reqs_thread_entry: start 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 20 get_system_obj_state: rctx=0x7ffcba42cfb8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55d57b149240 s->prefetch_data=0 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 20 Read xattr: user.rgw.idtag 2023-10-05T19:50:45.034 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.030+0000 7f593b7f7680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 10 removing key index: XDJQXDVKPUZOUVCAGMHV 2023-10-05T19:50:45.035 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f593b7f7680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.keys:XDJQXDVKPUZOUVCAGMHV, ofs0, ns] 2023-10-05T19:50:45.036 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.031+0000 7f592c8ed700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889697 cookie 94375382966048 notifier 4252 bl.length()=170 2023-10-05T19:50:45.036 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.033+0000 7f593b7f7680 10 removing email index: bar.client.0+test@test.test 2023-10-05T19:50:45.036 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.033+0000 7f593b7f7680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T19:50:45.036 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.033+0000 7f591ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889650 cookie 94375382955184 notifier 4252 bl.length()=178 2023-10-05T19:50:45.036 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.034+0000 7f593b7f7680 10 removing user buckets index 2023-10-05T19:50:45.037 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.034+0000 7f593b7f7680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-10-05T19:50:45.037 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.034+0000 7f592c8ed700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888151 cookie 94375382958288 notifier 4252 bl.length()=169 2023-10-05T19:50:45.038 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.036+0000 7f593b7f7680 10 removing user index: bar.client.0 2023-10-05T19:50:45.038 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.036+0000 7f593b7f7680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-10-05T19:50:45.038 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.036+0000 7f593b7f7680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:50:45.038 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.036+0000 7f591ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889651 cookie 94375382955184 notifier 4252 bl.length()=161 2023-10-05T19:50:45.042 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.040+0000 7f593b7f7680 20 remove_watcher() i=0 2023-10-05T19:50:45.042 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.040+0000 7f593b7f7680 2 removed watcher, disabling cache 2023-10-05T19:50:45.043 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.041+0000 7f593b7f7680 20 remove_watcher() i=1 2023-10-05T19:50:45.045 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.043+0000 7f593b7f7680 20 remove_watcher() i=2 2023-10-05T19:50:45.048 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.046+0000 7f593b7f7680 20 remove_watcher() i=3 2023-10-05T19:50:45.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.047+0000 7f593b7f7680 20 remove_watcher() i=4 2023-10-05T19:50:45.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.048+0000 7f593b7f7680 20 remove_watcher() i=5 2023-10-05T19:50:45.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.049+0000 7f593b7f7680 20 remove_watcher() i=6 2023-10-05T19:50:45.051 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.050+0000 7f593b7f7680 20 remove_watcher() i=7 2023-10-05T19:50:45.052 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.050+0000 7f58fffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:45.057 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-10-05T19:50:45.136 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:50:45.136 INFO:teuthology.orchestra.run.smithi172.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:50:45.142 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.140+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.143 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 realm 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.141+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.142+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:45.144 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.142+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.142+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:45.145 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.142+0000 7f7ce213e680 20 searching for the correct realm 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.152 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.149+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.153 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.154 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.150+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.156+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zonegroup_info.a35acc9d-0717-4920-871e-140f7de36a51 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.156+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zone_info.5cb8517b-2313-4992-9a76-073dfd099cc7 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.156+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.156+0000 7f7ce213e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.156+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:50:45.159 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 zone default found 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 4 Realm: () 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 4 ZoneGroup: default (a35acc9d-0717-4920-871e-140f7de36a51) 2023-10-05T19:50:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 4 Zone: default (5cb8517b-2313-4992-9a76-073dfd099cc7) 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 zonegroup default 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.157+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 started sync module instance, tier type = 2023-10-05T19:50:45.162 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.158+0000 7f7ce213e680 20 started zone id=5cb8517b-2313-4992-9a76-073dfd099cc7 (name=default) with tier type = 2023-10-05T19:50:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=0 2023-10-05T19:50:45.169 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=1 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=2 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=3 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=4 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=5 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.167+0000 7f7ce213e680 20 add_watcher() i=6 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.168+0000 7f7ce213e680 20 add_watcher() i=7 2023-10-05T19:50:45.170 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.168+0000 7f7ce213e680 2 all 8 watchers are set, enabling cache 2023-10-05T19:50:45.171 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.169+0000 7f7ca67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:45.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.169+0000 7f7ce213e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:50:45.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.169+0000 7f7ce213e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:50:45.218 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.214+0000 7f7c157fa700 20 reqs_thread_entry: start 2023-10-05T19:50:45.218 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.214+0000 7f7c0effd700 20 reqs_thread_entry: start 2023-10-05T19:50:45.218 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.214+0000 7f7ce213e680 20 init_complete bucket index max shards: 11 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7c0d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:51:15 2023 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7c0cff9700 20 reqs_thread_entry: start 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 20 get_system_obj_state: rctx=0x7fffc4a4ee68 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x560b0b415240 s->prefetch_data=0 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:50:45.219 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:45.220 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:50:45.220 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 20 Read xattr: user.rgw.idtag 2023-10-05T19:50:45.220 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 20 rados->read ofs=0 len=0 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.215+0000 7f7ce213e680 20 rados_obj.operate() r=0 bl.length=528 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7ce213e680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7ce213e680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7ce213e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7ce213e680 10 removing key index: IPLOTLGSDBZYLVCRYQHI 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7ce213e680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:IPLOTLGSDBZYLVCRYQHI, ofs0, ns] 2023-10-05T19:50:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.216+0000 7f7cceffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888152 cookie 94605435085792 notifier 4258 bl.length()=170 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.217+0000 7f7ce213e680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.217+0000 7f7ce213e680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.218+0000 7f7cce7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889698 cookie 94605435095888 notifier 4258 bl.length()=193 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.219+0000 7f7ce213e680 10 removing user buckets index 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.219+0000 7f7ce213e680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-10-05T19:50:45.223 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.219+0000 7f7cceffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889652 cookie 94605435097792 notifier 4258 bl.length()=184 2023-10-05T19:50:45.224 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.222+0000 7f7ce213e680 10 removing user index: testx$tenanteduser.client.0 2023-10-05T19:50:45.224 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.222+0000 7f7ce213e680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-10-05T19:50:45.224 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.222+0000 7f7ce213e680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:50:45.224 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.222+0000 7f7cce7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889699 cookie 94605435079296 notifier 4258 bl.length()=176 2023-10-05T19:50:45.229 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.227+0000 7f7ce213e680 20 remove_watcher() i=0 2023-10-05T19:50:45.229 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.227+0000 7f7ce213e680 2 removed watcher, disabling cache 2023-10-05T19:50:45.232 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.230+0000 7f7ce213e680 20 remove_watcher() i=1 2023-10-05T19:50:45.233 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.231+0000 7f7ce213e680 20 remove_watcher() i=2 2023-10-05T19:50:45.235 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.234+0000 7f7ce213e680 20 remove_watcher() i=3 2023-10-05T19:50:45.238 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.236+0000 7f7ce213e680 20 remove_watcher() i=4 2023-10-05T19:50:45.239 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.237+0000 7f7ce213e680 20 remove_watcher() i=5 2023-10-05T19:50:45.242 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.240+0000 7f7ce213e680 20 remove_watcher() i=6 2023-10-05T19:50:45.243 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.241+0000 7f7ce213e680 20 remove_watcher() i=7 2023-10-05T19:50:45.243 INFO:teuthology.orchestra.run.smithi172.stderr:2023-10-05T19:50:45.242+0000 7f7ca67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:50:45.248 INFO:tasks.s3tests:Removing s3-tests... 2023-10-05T19:50:45.249 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:50:45.405 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-10-05T19:50:45.414 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T19:50:45.424 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T19:50:45.424 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:45.425 INFO:tasks.rgw.client.0.smithi172.stdout:2023-10-05T19:50:45.423+0000 7fa14b069700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi172.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi172.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 90910) UID: 0 2023-10-05T19:50:45.425 INFO:tasks.rgw.client.0.smithi172.stdout:2023-10-05T19:50:45.423+0000 7fa15c344740 -1 shutting down 2023-10-05T19:50:50.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T19:50:51.526 INFO:tasks.rgw.client.0:Stopped 2023-10-05T19:50:51.526 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T19:50:51.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T19:50:51.624 DEBUG:teuthology.run_tasks:Unwinding manager vault 2023-10-05T19:50:51.634 INFO:tasks.vault:Stopping Vault instance 2023-10-05T19:50:51.635 DEBUG:tasks.vault.client.0:waiting for process to exit 2023-10-05T19:50:51.635 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:51.666 INFO:tasks.vault.client.0.smithi172.stdout:==> Vault shutdown triggered 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] core: marked as sealed 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] core: pre-seal teardown starting 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] rollback: stopping rollback manager 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] core: pre-seal teardown complete 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] core: stopping cluster listeners 2023-10-05T19:50:51.667 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:51.665Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2023-10-05T19:50:52.020 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:52.019Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2023-10-05T19:50:52.020 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:52.019Z [INFO] core: cluster listeners successfully shut down 2023-10-05T19:50:52.020 INFO:tasks.vault.client.0.smithi172.stderr:2023-10-05T19:50:52.019Z [INFO] core: vault is sealed 2023-10-05T19:50:52.021 INFO:tasks.vault.client.0:Stopped 2023-10-05T19:50:52.021 INFO:tasks.vault:Removing Vault... 2023-10-05T19:50:52.021 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2023-10-05T19:50:52.095 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T19:50:52.104 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T19:50:52.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:50:52.447 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-10-05T19:50:52.448 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":34,"fsid":"5e13a587-bd3a-4c44-ab2a-e881fd60ca16","created":"2023-10-05T19:45:38.001920+0000","modified":"2023-10-05T19:47:45.131230+0000","last_up_change":"2023-10-05T19:45:45.593853+0000","last_in_change":"2023-10-05T19:45:38.948574+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:45:45.913708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:45:46.630393+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T19:46:05.270299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T19:46:07.333775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T19:46:09.442023+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T19:46:11.054185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T19:46:13.040878+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T19:46:15.138828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-10-05T19:46:25.917688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-10-05T19:47:04.192299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4601c289-3389-47b7-b869-91f746bd4764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":86111},{"type":"v1","addr":"172.21.15.172:6804","nonce":86111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":86111},{"type":"v1","addr":"172.21.15.172:6809","nonce":86111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":86111},{"type":"v1","addr":"172.21.15.172:6815","nonce":86111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":86111},{"type":"v1","addr":"172.21.15.172:6813","nonce":86111}]},"public_addr":"172.21.15.172:6804/86111","cluster_addr":"172.21.15.172:6809/86111","heartbeat_back_addr":"172.21.15.172:6815/86111","heartbeat_front_addr":"172.21.15.172:6813/86111","state":["exists","up"]},{"osd":1,"uuid":"624131d9-ccda-440a-b258-3df6734b0890","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6801","nonce":86112},{"type":"v1","addr":"172.21.15.172:6800","nonce":86112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":86112},{"type":"v1","addr":"172.21.15.172:6805","nonce":86112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":86112},{"type":"v1","addr":"172.21.15.172:6811","nonce":86112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":86112},{"type":"v1","addr":"172.21.15.172:6808","nonce":86112}]},"public_addr":"172.21.15.172:6800/86112","cluster_addr":"172.21.15.172:6805/86112","heartbeat_back_addr":"172.21.15.172:6811/86112","heartbeat_front_addr":"172.21.15.172:6808/86112","state":["exists","up"]},{"osd":2,"uuid":"26fa6f73-54a9-4005-b2ee-02f42fe65f8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":86141},{"type":"v1","addr":"172.21.15.172:6817","nonce":86141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":86141},{"type":"v1","addr":"172.21.15.172:6819","nonce":86141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":86141},{"type":"v1","addr":"172.21.15.172:6823","nonce":86141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":86141},{"type":"v1","addr":"172.21.15.172:6821","nonce":86141}]},"public_addr":"172.21.15.172:6817/86141","cluster_addr":"172.21.15.172:6819/86141","heartbeat_back_addr":"172.21.15.172:6823/86141","heartbeat_front_addr":"172.21.15.172:6821/86141","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.249010+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.331326+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:45:43.316894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.31","mappings":[{"from":0,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.8","mappings":[{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.2b","mappings":[{"from":1,"to":2}]},{"pgid":"4.39","mappings":[{"from":1,"to":2}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"10.19","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:50:52.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T19:50:52.785 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T19:50:52.785 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T19:50:52.785 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T19:50:52.785 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:52.839 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T19:50:52.840 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T19:50:52.840 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:52.886 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T19:50:52.886 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T19:50:52.887 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:52.940 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T19:50:52.940 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T19:50:52.940 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T19:50:52.940 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:52.964 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T19:50:52.964 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T19:50:52.964 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T19:50:52.965 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:50:53.015 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T19:50:53.015 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T19:50:53.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:50:53.081 INFO:teuthology.orchestra.run.smithi172.stdout:1696535148.5996718 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:50:53.081 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T19:50:53.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:50:53.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:50:53.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:50:53.277 INFO:teuthology.orchestra.run.smithi172.stdout:1696535148.5996718 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:50:53.277 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:50:53.277 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T19:50:55.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T19:50:55.718 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:50:55.718 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T19:50:55.807 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:50:55.807 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T19:50:55.880 INFO:tasks.ceph:Archiving mon data... 2023-10-05T19:50:55.881 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761/data/mon.a.tgz 2023-10-05T19:50:55.882 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2023-10-05T19:50:55.896 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.62X9GvWbO6 2023-10-05T19:50:55.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.62X9GvWbO6 2023-10-05T19:50:56.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.62X9GvWbO6 2023-10-05T19:50:56.113 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.62X9GvWbO6 is 272KB 2023-10-05T19:50:56.208 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.62X9GvWbO6 2023-10-05T19:50:56.221 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T19:50:56.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T19:50:56.284 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T19:50:56.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T19:50:56.349 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T19:50:56.355 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T19:50:56.357 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761/remote/smithi172/crash 2023-10-05T19:50:56.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T19:50:56.418 INFO:tasks.ceph:Compressing logs... 2023-10-05T19:50:56.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:51:24.569 INFO:tasks.ceph:Archiving logs... 2023-10-05T19:51:24.570 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761/remote/smithi172/log 2023-10-05T19:51:24.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T19:51:27.875 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T19:51:27.884 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T19:51:27.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T19:51:27.940 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T19:51:27.949 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T19:51:27.949 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:51:27.970 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-10-05T19:51:27.973 INFO:teuthology.orchestra.run.smithi172.stdout:210 Number of sources = 4 2023-10-05T19:51:27.973 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T19:51:27.973 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-10-05T19:51:27.974 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 -901us[ -901us] +/- 91ms 2023-10-05T19:51:27.974 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -360us[ -359us] +/- 67ms 2023-10-05T19:51:27.974 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +1342us[+1342us] +/- 88ms 2023-10-05T19:51:27.974 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:51:27.974 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T19:51:27.983 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T19:51:27.984 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T19:51:27.993 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-05T19:51:28.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T19:51:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:51:28.132 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T19:51:28.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T19:51:28.157 INFO:teuthology.task.internal:Duration was 1106.693175 seconds 2023-10-05T19:51:28.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T19:51:28.167 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T19:51:28.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T19:51:28.215 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T19:51:28.625 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T19:51:28.625 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-10-05T19:51:28.626 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T19:51:28.683 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T19:51:28.684 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:51:28.762 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T19:51:28.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T19:51:28.875 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T19:51:28.886 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T19:51:28.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T19:51:28.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T19:51:28.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:51:28.974 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-10-05T19:51:28.989 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:51:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:51:29.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T19:51:29.054 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T19:51:29.054 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761/remote/smithi172 2023-10-05T19:51:29.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T19:51:29.119 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T19:51:29.120 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T19:51:29.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T19:51:29.177 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T19:51:29.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T19:51:29.186 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T19:51:29.186 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T19:51:29.220 INFO:teuthology.orchestra.run.smithi172.stdout: 262255 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 19:51 /home/ubuntu/cephtest 2023-10-05T19:51:29.221 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T19:51:29.231 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T19:51:29.250 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T19:51:29.276 DEBUG:teuthology.nuke:shortname: smithi172 2023-10-05T19:51:29.277 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:51:29.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:25:50.409662', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvnFQtyyPRYWPgri6uc34sDcJemY45OMhSqNvFCy5pir0u3AHWCvD56dzwRnNfAz+Y0TckFOlVUYp/kjtEG/EA='} 2023-10-05T19:51:29.348 INFO:teuthology.orchestra.console.smithi172:Power off 2023-10-05T19:51:29.348 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T19:51:29.375 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-10-05T19:51:29.912 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:51:29.936 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-05T19:51:34.090 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:51:34.222 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-10-05T19:51:38.270 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:51:38.380 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-10-05T19:51:38.481 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-10-05T19:51:38.585 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} duration: 1106.6931748390198 failure_reason: '"1696535148.5996718 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T19:51:38.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:51:38.677 INFO:teuthology.run:FAIL