2023-10-05T19:42:59.040 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:42:59.040 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769 -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769/orig.config.yaml 2023-10-05T19:42:59.082 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:42:59.157 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} email: yweinste@redhat.com first_in_suite: false job_id: '7411769' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt vault auth: token rgw crypt vault prefix: /v1/transit/export/encryption-key/ rgw crypt vault secret engine: transit setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 rgw: client.0: use-vault-role: client.0 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYiwqWwZ6dyEyEwETysEVt7X3DeM0X0sKB7EP5gkcGqxrvp1es/k5VfSSPvll5vzEXXKIz+V56EoZwhgmO4OoI= tasks: - install: null - ceph: null - vault: client.0: engine: transit flavor: old install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/transit/keys/ root_token: test_root_token secrets: - path: my-key-1 - path: my-key-2 - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:42:59.164 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:42:59.165 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:42:59.165 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:42:59.166 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:42:59.189 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:42:59.189 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:42:59.189 INFO:teuthology.packaging:ref: None 2023-10-05T19:42:59.189 INFO:teuthology.packaging:tag: None 2023-10-05T19:42:59.189 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:42:59.189 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:42:59.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:42:59.337 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278.g487ffcb5 2023-10-05T19:42:59.337 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:42:59.343 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:42:59.343 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:42:59.347 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:42:59.359 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:42:59.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:42:59.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:37:44.724970', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYiwqWwZ6dyEyEwETysEVt7X3DeM0X0sKB7EP5gkcGqxrvp1es/k5VfSSPvll5vzEXXKIz+V56EoZwhgmO4OoI='} 2023-10-05T19:42:59.386 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:42:59.391 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-05T19:42:59.391 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:42:59.428 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f019c2b6ee0>, signals=[15]) 2023-10-05T19:42:59.706 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:42:59.711 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:42:59.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T19:42:59.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:42:59.781 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:42:59.795 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2023-10-05T19:42:59.812 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2023-10-05T19:42:59.812 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2023-10-05T19:42:59.867 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux" 2023-10-05T19:42:59.867 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8.4 (Ootpa)" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8.4" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:42:59.868 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-05T19:42:59.869 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-05T19:42:59.869 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-05T19:42:59.869 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-05T19:42:59.869 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2023-10-05T19:42:59.903 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:42:59.910 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:42:59.915 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:42:59.915 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:42:59.931 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:42:59.953 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:42:59.954 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:42:59.987 INFO:teuthology.orchestra.run.smithi155.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-05T19:42:59.988 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:43:00.022 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:43:00.037 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:43:00.037 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:43:00.037 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:43:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:43:00.053 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2023-10-05T19:43:00.110 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-05T19:43:00.110 INFO:teuthology.task.kernel:Running kernel on smithi155: 4.18.0-305.25.1.el8_4.x86_64 2023-10-05T19:43:00.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2023-10-05T19:43:01.590 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:01.590 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2023-10-05T19:43:02.983 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-05T19:43:03.612 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2023-10-05T19:43:07.067 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2023-10-05T19:43:07.753 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-05T19:43:07.753 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-05T19:43:07.753 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-05T19:43:07.915 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:43:07.916 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:43:07.916 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:43:08.030 DEBUG:teuthology.orchestra.run.smithi155:> echo no | sudo yum reinstall kernel || true 2023-10-05T19:43:08.376 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:08.376 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2023-10-05T19:43:08.665 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:01 ago on Thu 05 Oct 2023 07:43:07 PM UTC. 2023-10-05T19:43:08.783 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-05T19:43:08.783 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:43:08.783 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:43:08.816 INFO:teuthology.orchestra.run.smithi155.stderr:Error: No packages marked for reinstall. 2023-10-05T19:43:08.867 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum reinstall -y kernel || true 2023-10-05T19:43:09.211 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:43:09.211 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2023-10-05T19:43:09.501 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:02 ago on Thu 05 Oct 2023 07:43:07 PM UTC. 2023-10-05T19:43:09.639 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-05T19:43:09.640 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:43:09.640 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-05T19:43:09.682 INFO:teuthology.orchestra.run.smithi155.stderr:Error: No packages marked for reinstall. 2023-10-05T19:43:09.738 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:43:09.863 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:09.863 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:09.863 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:09.863 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:43:09.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:43:09.864 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:43:09.864 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:43:09.864 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2023-10-05T19:43:09.989 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:09.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:09.990 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2023-10-05T19:43:10.016 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.02 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 99.el8_4.1 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Group : System Environment/Base 2023-10-05T19:43:10.017 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 9264631 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Relocations : (not relocatable) 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : Red Hat, Inc. 2023-10-05T19:43:10.018 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : Red Hat, Inc. 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:43:10.019 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2023-10-05T19:43:10.020 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-05T19:43:10.021 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:10.021 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-05T19:43:11.470 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2023-10-05T19:43:13.569 INFO:teuthology.orchestra.run.smithi155.stderr:done 2023-10-05T19:43:13.571 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2023-10-05T19:43:13.614 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.RDw8mcCozi 2023-10-05T19:43:13.615 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RDw8mcCozi 2023-10-05T19:43:13.718 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.RDw8mcCozi 2023-10-05T19:43:13.828 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.RDw8mcCozi is 6KB 2023-10-05T19:43:13.876 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.RDw8mcCozi 2023-10-05T19:43:13.889 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2023-10-05T19:43:13.960 INFO:teuthology.orchestra.run.smithi155.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-05T19:43:13.960 INFO:teuthology.orchestra.run.smithi155.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-05T19:43:13.960 INFO:teuthology.orchestra.run.smithi155.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-05T19:43:13.961 INFO:teuthology.orchestra.run.smithi155.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-05T19:43:13.961 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-05T19:43:14.106 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2023-10-05T19:43:44.138 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:43:44.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:43:44.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:44:02.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2023-10-05T19:44:11.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:44:11.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:44:14.746 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2023-10-05T19:44:26.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:44:26.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:45:26.801 DEBUG:teuthology.orchestra.remote:timed out 2023-10-05T19:45:41.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:45:41.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:45:42.229 DEBUG:teuthology.orchestra.run.smithi155:> true 2023-10-05T19:45:42.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:45:42.470 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-05T19:45:42.470 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2023-10-05T19:45:42.487 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-05T19:45:42.487 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-05T19:45:42.487 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:45:42.487 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-05T19:45:43.489 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T19:45:43.489 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:45:43.594 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2023-10-05T19:45:43.607 DEBUG:teuthology.parallel:result is None 2023-10-05T19:45:43.607 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T19:45:43.616 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T19:45:43.616 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:45:43.632 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T19:45:43.651 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T19:45:43.658 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T19:45:43.658 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:45:43.708 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T19:45:43.715 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T19:45:43.715 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:45:43.785 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:45:43.795 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:45:43.796 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T19:45:43.807 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T19:45:43.807 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:45:43.862 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T19:45:43.871 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T19:45:43.871 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:45:43.918 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:45:44.004 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:45:44.040 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:45:44.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:45:44.161 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:45:44.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:45:44.225 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2023-10-05T19:45:44.309 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T19:45:44.475 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T19:45:44.483 INFO:teuthology.task.internal:Starting timer... 2023-10-05T19:45:44.483 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T19:45:44.492 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T19:45:44.501 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-05T19:45:44.501 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2023-10-05T19:45:44.566 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: [ OK ] 2023-10-05T19:45:44.568 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T19:45:44.568 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2023-10-05T19:45:44.590 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2023-10-05T19:45:44.590 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2023-10-05T19:45:44.590 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T19:45:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:45:44.658 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T19:45:44.658 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2023-10-05T19:45:44.725 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T19:45:44.735 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-05T19:45:44.741 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-05T19:45:44.862 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T19:45:44.876 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T19:45:44.877 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T19:53:42.518 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2023-10-05T19:53:42.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:53:42.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:53:42.590 DEBUG:teuthology.orchestra.run.smithi155:> true 2023-10-05T19:53:42.652 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-10-05T19:53:42.652 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T19:53:42.663 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T19:53:42.663 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:53:42.663 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:53:42.720 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-05T19:53:42.734 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-05T19:53:42.770 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2023-10-05T19:53:42.780 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2023-10-05T19:53:42.794 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-05T19:53:42.808 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-05T19:53:42.876 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:210 Number of sources = 4 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:53:42.878 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T19:53:42.879 INFO:teuthology.run_tasks:Running task install... 2023-10-05T19:53:42.887 DEBUG:teuthology.task.install:project ceph 2023-10-05T19:53:42.887 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T19:53:42.888 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T19:53:42.888 INFO:teuthology.task.install:Using flavor: default 2023-10-05T19:53:42.893 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T19:53:42.893 INFO:teuthology.task.install:extra packages: [] 2023-10-05T19:53:42.894 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'tag': None, 'wait_for_package': False} 2023-10-05T19:53:42.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:53:43.112 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/ 2023-10-05T19:53:43.112 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1278.g487ffcb5 2023-10-05T19:53:43.276 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-05T19:53:43.276 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:53:43.277 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-05T19:53:43.302 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-05T19:53:43.303 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/487ffcb5ddb60d29c5688d1a457e812b1b697b90/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-05T19:53:43.369 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-05T19:53:43.444 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-05T19:53:43.509 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2023-10-05T19:53:43.510 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2023-10-05T19:53:43.901 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:53:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:44 files removed 2023-10-05T19:53:44.287 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2023-10-05T19:53:44.637 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:53:45.455 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 257 kB/s | 78 kB 00:00 2023-10-05T19:53:45.825 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2023-10-05T19:53:46.158 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 4.8 kB/s | 1.4 kB 00:00 2023-10-05T19:53:46.297 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for python3-asyncssh owned by ceph 32 kB/s | 3.5 kB 00:00 2023-10-05T19:53:47.888 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-10-05T19:53:51.203 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-05T19:53:51.572 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 33 MB 00:00 2023-10-05T19:53:56.718 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 125 MB/s | 38 MB 00:00 2023-10-05T19:54:05.584 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:54:05.586 INFO:teuthology.orchestra.run.smithi155.stdout:==================================================================================================== 2023-10-05T19:54:05.586 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:54:05.586 INFO:teuthology.orchestra.run.smithi155.stdout:==================================================================================================== 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.6 M 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.2 M 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 23 M 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 24 k 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-05T19:54:05.587 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 797 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 506 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 5.6 M 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 45 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 115 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 191 k 2023-10-05T19:54:05.588 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 370 k 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 359 k 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 106 k 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout:==================================================================================================== 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout:Install 21 Packages 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2023-10-05T19:54:05.589 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:05.590 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 63 M 2023-10-05T19:54:05.590 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:54:06.209 INFO:teuthology.orchestra.run.smithi155.stdout:(1/23): ceph-base-17.2.6-1278.g487ffcb5.el8.x86 11 MB/s | 5.2 MB 00:00 2023-10-05T19:54:06.278 INFO:teuthology.orchestra.run.smithi155.stdout:(2/23): ceph-selinux-17.2.6-1278.g487ffcb5.el8. 357 kB/s | 24 kB 00:00 2023-10-05T19:54:06.424 INFO:teuthology.orchestra.run.smithi155.stdout:(3/23): libcephfs2-17.2.6-1278.g487ffcb5.el8.x8 5.3 MB/s | 797 kB 00:00 2023-10-05T19:54:06.558 INFO:teuthology.orchestra.run.smithi155.stdout:(4/23): libradosstriper1-17.2.6-1278.g487ffcb5. 3.7 MB/s | 506 kB 00:00 2023-10-05T19:54:07.017 INFO:teuthology.orchestra.run.smithi155.stdout:(5/23): librgw2-17.2.6-1278.g487ffcb5.el8.x86_6 12 MB/s | 5.6 MB 00:00 2023-10-05T19:54:07.103 INFO:teuthology.orchestra.run.smithi155.stdout:(6/23): python3-ceph-argparse-17.2.6-1278.g487f 521 kB/s | 45 kB 00:00 2023-10-05T19:54:07.145 INFO:teuthology.orchestra.run.smithi155.stdout:(7/23): python3-ceph-common-17.2.6-1278.g487ffc 2.7 MB/s | 115 kB 00:00 2023-10-05T19:54:07.338 INFO:teuthology.orchestra.run.smithi155.stdout:(8/23): ceph-common-17.2.6-1278.g487ffcb5.el8.x 15 MB/s | 23 MB 00:01 2023-10-05T19:54:07.340 INFO:teuthology.orchestra.run.smithi155.stdout:(9/23): python3-cephfs-17.2.6-1278.g487ffcb5.el 976 kB/s | 191 kB 00:00 2023-10-05T19:54:07.382 INFO:teuthology.orchestra.run.smithi155.stdout:(10/23): python3-rados-17.2.6-1278.g487ffcb5.el 8.3 MB/s | 370 kB 00:00 2023-10-05T19:54:07.393 INFO:teuthology.orchestra.run.smithi155.stdout:(11/23): python3-rbd-17.2.6-1278.g487ffcb5.el8. 6.7 MB/s | 359 kB 00:00 2023-10-05T19:54:07.405 INFO:teuthology.orchestra.run.smithi155.stdout:(12/23): python3-rgw-17.2.6-1278.g487ffcb5.el8. 4.6 MB/s | 106 kB 00:00 2023-10-05T19:54:07.691 INFO:teuthology.orchestra.run.smithi155.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 206 kB/s | 59 kB 00:00 2023-10-05T19:54:07.783 INFO:teuthology.orchestra.run.smithi155.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 817 kB/s | 75 kB 00:00 2023-10-05T19:54:07.813 INFO:teuthology.orchestra.run.smithi155.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 730 kB/s | 306 kB 00:00 2023-10-05T19:54:07.844 INFO:teuthology.orchestra.run.smithi155.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2023-10-05T19:54:07.861 INFO:teuthology.orchestra.run.smithi155.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-10-05T19:54:07.879 INFO:teuthology.orchestra.run.smithi155.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 20 MB/s | 353 kB 00:00 2023-10-05T19:54:07.892 INFO:teuthology.orchestra.run.smithi155.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.4 MB/s | 47 kB 00:00 2023-10-05T19:54:08.089 INFO:teuthology.orchestra.run.smithi155.stdout:(20/23): librados2-17.2.6-1278.g487ffcb5.el8.x8 18 MB/s | 3.5 MB 00:00 2023-10-05T19:54:08.094 INFO:teuthology.orchestra.run.smithi155.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2023-10-05T19:54:08.242 INFO:teuthology.orchestra.run.smithi155.stdout:(22/23): librbd1-17.2.6-1278.g487ffcb5.el8.x86_ 23 MB/s | 3.6 MB 00:00 2023-10-05T19:54:12.021 INFO:teuthology.orchestra.run.smithi155.stdout:(23/23): ceph-radosgw-17.2.6-1278.g487ffcb5.el8 2.7 MB/s | 17 MB 00:06 2023-10-05T19:54:12.027 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:54:12.027 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.8 MB/s | 63 MB 00:06 2023-10-05T19:54:12.032 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:54:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:54:12.080 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:54:12.873 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:54:12.877 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:54:13.975 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:54:14.153 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-05T19:54:14.540 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-10-05T19:54:14.744 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:54:14.847 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:54:15.051 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T19:54:15.346 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-10-05T19:54:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:54:15.961 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:54:16.178 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T19:54:16.260 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/25 2023-10-05T19:54:16.443 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-10-05T19:54:16.620 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-05T19:54:16.839 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-10-05T19:54:17.497 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-10-05T19:54:17.686 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:54:17.760 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:54:17.897 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 11/25 2023-10-05T19:54:18.123 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T19:54:18.346 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T19:54:18.556 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 14/25 2023-10-05T19:54:18.735 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T19:54:18.807 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 15/25 2023-10-05T19:54:18.987 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-10-05T19:54:19.189 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-10-05T19:54:19.415 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-10-05T19:54:19.650 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 19/25 2023-10-05T19:54:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:54:24.894 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:54:25.477 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 20/25 2023-10-05T19:54:26.235 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T19:54:26.317 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 21/25 2023-10-05T19:54:26.434 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:54:44.511 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:54:44.511 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2023-10-05T19:54:44.511 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2023-10-05T19:54:44.511 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2023-10-05T19:54:44.511 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /run 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2023-10-05T19:54:44.512 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T19:54:46.357 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 23/25 2023-10-05T19:54:46.358 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T19:54:46.567 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-05T19:54:46.567 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:54:47.392 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:54:47.392 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/25 2023-10-05T19:54:47.392 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/25 2023-10-05T19:54:47.392 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_6 6/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 7/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8. 8/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x8 9/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 10/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 11/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 12/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 13/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-05T19:54:47.393 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 22/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-05T19:54:47.394 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/25 2023-10-05T19:54:48.904 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-05T19:54:48.904 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:52 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:54:50.863 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:54:50.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:50.975 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2023-10-05T19:54:50.975 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.975 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.975 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.976 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:54:50.977 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-05T19:54:50.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:50.978 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:54:51.122 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2023-10-05T19:54:51.491 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:54:52.372 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:56 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:54:54.181 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 52 M 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-05T19:54:54.182 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 174 k 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2023-10-05T19:54:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:54:54.184 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 53 M 2023-10-05T19:54:54.184 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 233 M 2023-10-05T19:54:54.184 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:54:54.188 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-10-05T19:54:54.225 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.3 MB/s | 161 kB 00:00 2023-10-05T19:54:54.241 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-10-05T19:54:54.257 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-10-05T19:54:54.456 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): libcephsqlite-17.2.6-1278.g487ffcb5.el8. 641 kB/s | 174 kB 00:00 2023-10-05T19:54:56.731 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-17.2.6-1278.g487ffcb5.el8.x86_ 20 MB/s | 52 MB 00:02 2023-10-05T19:54:56.734 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:54:56.734 INFO:teuthology.orchestra.run.smithi155.stdout:Total 21 MB/s | 53 MB 00:02 2023-10-05T19:54:56.738 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:54:56.760 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:54:56.760 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:54:57.129 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:54:57.133 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:54:57.635 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:54:57.817 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-10-05T19:54:58.030 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-05T19:54:58.108 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-10-05T19:54:58.274 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-10-05T19:54:58.462 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-05T19:54:58.613 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T19:55:03.610 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/6 2023-10-05T19:55:03.844 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T19:55:04.111 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 6/6 2023-10-05T19:55:04.111 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/6 2023-10-05T19:55:04.111 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/6 2023-10-05T19:55:04.112 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-05T19:55:04.112 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-10-05T19:55:04.112 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-05T19:55:05.177 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-10-05T19:55:05.177 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:09 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:07.031 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.5-12.el8.x86_64 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-05T19:55:07.128 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-05T19:55:07.129 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-05T19:55:07.129 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:07.129 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:07.272 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2023-10-05T19:55:07.630 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:08.477 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:12 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:10.366 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:10.368 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================= 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================= 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 6.4 k 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 2.3 M 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 1.6 M 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 238 k 2023-10-05T19:55:10.369 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.9 M 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 17 M 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-05T19:55:10.370 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-05T19:55:10.371 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================= 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout:Install 28 Packages 2023-10-05T19:55:10.372 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:10.373 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 27 M 2023-10-05T19:55:10.373 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 106 M 2023-10-05T19:55:10.373 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:55:10.628 INFO:teuthology.orchestra.run.smithi155.stdout:(1/28): ceph-17.2.6-1278.g487ffcb5.el8.x86_64.r 70 kB/s | 6.4 kB 00:00 2023-10-05T19:55:10.824 INFO:teuthology.orchestra.run.smithi155.stdout:(2/28): ceph-mds-17.2.6-1278.g487ffcb5.el8.x86_ 8.0 MB/s | 2.3 MB 00:00 2023-10-05T19:55:11.091 INFO:teuthology.orchestra.run.smithi155.stdout:(3/28): ceph-mon-17.2.6-1278.g487ffcb5.el8.x86_ 8.5 MB/s | 3.9 MB 00:00 2023-10-05T19:55:11.157 INFO:teuthology.orchestra.run.smithi155.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1278.g487f 3.6 MB/s | 238 kB 00:00 2023-10-05T19:55:11.665 INFO:teuthology.orchestra.run.smithi155.stdout:(5/28): ceph-osd-17.2.6-1278.g487ffcb5.el8.x86_ 20 MB/s | 17 MB 00:00 2023-10-05T19:55:11.666 INFO:teuthology.orchestra.run.smithi155.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 86 kB/s | 44 kB 00:00 2023-10-05T19:55:11.705 INFO:teuthology.orchestra.run.smithi155.stdout:(7/28): ceph-mgr-17.2.6-1278.g487ffcb5.el8.x86_ 1.3 MB/s | 1.6 MB 00:01 2023-10-05T19:55:11.986 INFO:teuthology.orchestra.run.smithi155.stdout:(8/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 539 kB/s | 173 kB 00:00 2023-10-05T19:55:12.087 INFO:teuthology.orchestra.run.smithi155.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-10-05T19:55:12.190 INFO:teuthology.orchestra.run.smithi155.stdout:(10/28): python3-jaraco-functools-2.0-4.el8.noa 172 kB/s | 18 kB 00:00 2023-10-05T19:55:12.305 INFO:teuthology.orchestra.run.smithi155.stdout:(11/28): python3-beautifulsoup4-4.6.3-2.el8.1.n 290 kB/s | 185 kB 00:00 2023-10-05T19:55:12.312 INFO:teuthology.orchestra.run.smithi155.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 398 kB/s | 49 kB 00:00 2023-10-05T19:55:12.422 INFO:teuthology.orchestra.run.smithi155.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 502 kB/s | 59 kB 00:00 2023-10-05T19:55:12.458 INFO:teuthology.orchestra.run.smithi155.stdout:(14/28): python3-cherrypy-18.4.0-1.el8.noarch.r 511 kB/s | 384 kB 00:00 2023-10-05T19:55:12.516 INFO:teuthology.orchestra.run.smithi155.stdout:(15/28): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-10-05T19:55:12.523 INFO:teuthology.orchestra.run.smithi155.stdout:(16/28): python3-portend-2.6-1.el8.noarch.rpm 154 kB/s | 16 kB 00:00 2023-10-05T19:55:12.559 INFO:teuthology.orchestra.run.smithi155.stdout:(17/28): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-10-05T19:55:12.618 INFO:teuthology.orchestra.run.smithi155.stdout:(18/28): python3-singledispatch-3.4.0.3-18.el8. 236 kB/s | 24 kB 00:00 2023-10-05T19:55:12.630 INFO:teuthology.orchestra.run.smithi155.stdout:(19/28): python3-tempora-1.14.1-5.el8.noarch.rp 274 kB/s | 29 kB 00:00 2023-10-05T19:55:12.661 INFO:teuthology.orchestra.run.smithi155.stdout:(20/28): python3-trustme-0.6.0-4.el8.noarch.rpm 270 kB/s | 27 kB 00:00 2023-10-05T19:55:12.749 INFO:teuthology.orchestra.run.smithi155.stdout:(21/28): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2023-10-05T19:55:12.777 INFO:teuthology.orchestra.run.smithi155.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 736 kB/s | 85 kB 00:00 2023-10-05T19:55:12.814 INFO:teuthology.orchestra.run.smithi155.stdout:(23/28): python3-werkzeug-0.12.2-4.el8.noarch.r 12 MB/s | 457 kB 00:00 2023-10-05T19:55:12.818 INFO:teuthology.orchestra.run.smithi155.stdout:(24/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-10-05T19:55:12.832 INFO:teuthology.orchestra.run.smithi155.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 2.1 MB/s | 39 kB 00:00 2023-10-05T19:55:12.848 INFO:teuthology.orchestra.run.smithi155.stdout:(26/28): python3-zc-lockfile-2.0-2.el8.noarch.r 232 kB/s | 23 kB 00:00 2023-10-05T19:55:12.849 INFO:teuthology.orchestra.run.smithi155.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 3.3 MB/s | 54 kB 00:00 2023-10-05T19:55:12.850 INFO:teuthology.orchestra.run.smithi155.stdout:(28/28): python3-mako-1.0.6-13.el8.noarch.rpm 4.8 MB/s | 157 kB 00:00 2023-10-05T19:55:12.858 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:55:12.858 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 27 MB 00:02 2023-10-05T19:55:12.863 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:55:12.901 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:55:12.902 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:55:13.295 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:55:13.300 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:55:14.340 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:55:14.561 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-10-05T19:55:14.792 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-10-05T19:55:14.965 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-10-05T19:55:15.210 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-10-05T19:55:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-10-05T19:55:15.794 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-10-05T19:55:16.006 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-10-05T19:55:16.244 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-10-05T19:55:16.401 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-10-05T19:55:16.565 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-10-05T19:55:16.781 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-10-05T19:55:16.963 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-10-05T19:55:17.083 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-10-05T19:55:17.318 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-10-05T19:55:17.514 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-10-05T19:55:17.708 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-10-05T19:55:17.960 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-10-05T19:55:18.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-10-05T19:55:18.413 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-10-05T19:55:18.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-10-05T19:55:19.687 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-10-05T19:55:19.975 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-10-05T19:55:20.359 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 23/28 2023-10-05T19:55:20.527 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/28 2023-10-05T19:55:22.022 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 24/28 2023-10-05T19:55:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 25/28 2023-10-05T19:55:22.590 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 25/28 2023-10-05T19:55:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 26/28 2023-10-05T19:55:22.987 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 26/28 2023-10-05T19:55:23.134 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 27/28 2023-10-05T19:55:23.153 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 27/28 2023-10-05T19:55:23.259 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 28/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 3/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 4/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 5/28 2023-10-05T19:55:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8. 6/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-10-05T19:55:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-10-05T19:55:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-10-05T19:55:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-10-05T19:55:25.167 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:29 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:27.072 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:55:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:27.173 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:55:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-05T19:55:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-05T19:55:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-2.6-1.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-05T19:55:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-05T19:55:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:27.178 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:27.316 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2023-10-05T19:55:27.684 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:28.512 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:29.829 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:55:30.310 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:30.312 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:55:30.312 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:30.843 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2023-10-05T19:55:31.207 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:32.114 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:36 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:34.022 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:34.023 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:34.023 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:55:34.023 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:34.023 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:55:34.023 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 80 k 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 80 k 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 364 k 2023-10-05T19:55:34.024 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:55:34.162 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-17.2.6-1278.g487ffcb5.el8.noarch.rpm 587 kB/s | 80 kB 00:00 2023-10-05T19:55:34.162 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:55:34.162 INFO:teuthology.orchestra.run.smithi155.stdout:Total 581 kB/s | 80 kB 00:00 2023-10-05T19:55:34.162 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:55:34.169 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:55:34.169 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:55:34.171 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:55:34.175 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:55:34.222 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:55:37.530 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:55:37.742 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:55:37.948 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:55:38.966 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:55:38.967 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:40.878 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:55:40.973 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:40.973 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:55:40.974 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:55:40.974 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:41.150 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2023-10-05T19:55:41.525 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:42.440 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:44.229 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 162 k 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:55:44.230 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:55:44.231 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:44.231 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 162 k 2023-10-05T19:55:44.231 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 490 k 2023-10-05T19:55:44.231 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:55:44.396 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-17.2.6-1278.g487ffc 981 kB/s | 162 kB 00:00 2023-10-05T19:55:44.396 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:55:44.397 INFO:teuthology.orchestra.run.smithi155.stdout:Total 972 kB/s | 162 kB 00:00 2023-10-05T19:55:44.397 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:55:44.411 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:55:44.411 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:55:44.463 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:55:44.467 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:55:44.584 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:55:44.731 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:55:45.164 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:55:46.222 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5. 1/1 2023-10-05T19:55:46.223 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:50 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:48.179 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:55:48.277 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:48.278 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:55:48.278 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:55:48.278 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:48.278 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:48.417 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2023-10-05T19:55:48.784 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:49.655 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:53 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:51.009 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:55:51.498 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:51.499 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:55:51.499 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:55:51.611 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2023-10-05T19:55:51.973 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:55:52.898 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:56 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:55:54.812 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout:=================================================================================================== 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout:=================================================================================================== 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 1.7 M 2023-10-05T19:55:54.813 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 23 k 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 14 k 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout:=================================================================================================== 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2023-10-05T19:55:54.814 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:55:54.815 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2023-10-05T19:55:54.815 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 8.0 M 2023-10-05T19:55:54.815 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:55:55.077 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1278.g487 204 kB/s | 23 kB 00:00 2023-10-05T19:55:55.134 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 586 kB/s | 34 kB 00:00 2023-10-05T19:55:55.152 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): ceph-prometheus-alerts-17.2.6-1278.g487f 77 kB/s | 14 kB 00:00 2023-10-05T19:55:55.189 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2023-10-05T19:55:55.209 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1278.g487ffcb5 6.9 MB/s | 1.7 MB 00:00 2023-10-05T19:55:55.217 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.3 MB/s | 196 kB 00:00 2023-10-05T19:55:55.220 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:55:55.220 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.9 MB/s | 2.0 MB 00:00 2023-10-05T19:55:55.222 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:55:55.230 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:55:55.230 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:55:55.316 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:55:55.321 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:55:55.588 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:55:55.807 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-05T19:55:55.979 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-05T19:55:56.182 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-05T19:55:56.293 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 4/6 2023-10-05T19:55:56.676 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 5/6 2023-10-05T19:55:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T19:55:57.171 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 6/6 2023-10-05T19:55:57.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8. 1/6 2023-10-05T19:55:57.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarc 2/6 2023-10-05T19:55:57.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.n 3/6 2023-10-05T19:55:57.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-05T19:55:57.172 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-05T19:55:58.232 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-05T19:55:58.232 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:02 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:00.262 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:00.359 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:00.360 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:00.475 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-05T19:56:00.845 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:01.697 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:05 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:03.549 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 7.4 M 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-05T19:56:03.551 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout:Install 8 Packages 2023-10-05T19:56:03.552 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:03.553 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 36 M 2023-10-05T19:56:03.553 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 199 M 2023-10-05T19:56:03.553 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:03.592 INFO:teuthology.orchestra.run.smithi155.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 5.6 MB/s | 225 kB 00:00 2023-10-05T19:56:03.659 INFO:teuthology.orchestra.run.smithi155.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 56 MB/s | 3.7 MB 00:00 2023-10-05T19:56:03.741 INFO:teuthology.orchestra.run.smithi155.stdout:(3/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 58 MB/s | 4.7 MB 00:00 2023-10-05T19:56:03.857 INFO:teuthology.orchestra.run.smithi155.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 48 MB/s | 15 MB 00:00 2023-10-05T19:56:03.873 INFO:teuthology.orchestra.run.smithi155.stdout:(5/8): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-10-05T19:56:03.897 INFO:teuthology.orchestra.run.smithi155.stdout:(6/8): libgfortran-8.4.1-1.el8.x86_64.rpm 27 MB/s | 639 kB 00:00 2023-10-05T19:56:03.915 INFO:teuthology.orchestra.run.smithi155.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 26 MB/s | 4.6 MB 00:00 2023-10-05T19:56:04.092 INFO:teuthology.orchestra.run.smithi155.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-127 14 MB/s | 7.4 MB 00:00 2023-10-05T19:56:04.093 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:04.094 INFO:teuthology.orchestra.run.smithi155.stdout:Total 66 MB/s | 36 MB 00:00 2023-10-05T19:56:04.103 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:04.122 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:04.122 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:04.565 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:04.570 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:05.179 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:05.353 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-05T19:56:05.524 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-10-05T19:56:05.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-05T19:56:06.227 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-10-05T19:56:06.411 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-05T19:56:06.958 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-10-05T19:56:07.144 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-10-05T19:56:07.873 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-10-05T19:56:08.162 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-10-05T19:56:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-05T19:56:10.461 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-10-05T19:56:11.744 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-10-05T19:56:11.918 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T19:56:12.222 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 8/8 2023-10-05T19:56:12.222 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb 1/8 2023-10-05T19:56:12.222 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-10-05T19:56:12.222 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-10-05T19:56:12.222 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-10-05T19:56:12.223 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-10-05T19:56:12.223 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-10-05T19:56:12.223 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-10-05T19:56:13.256 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-10-05T19:56:13.256 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:17 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:15.191 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:15.292 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:15.292 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:15.292 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:15.293 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:15.439 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2023-10-05T19:56:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:16.646 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:20 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:18.514 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================== 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================== 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 47 k 2023-10-05T19:56:18.516 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-05T19:56:18.517 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================================== 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2023-10-05T19:56:18.518 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:18.519 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.9 M 2023-10-05T19:56:18.519 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 25 M 2023-10-05T19:56:18.519 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:19.089 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 79 kB/s | 16 kB 00:00 2023-10-05T19:56:19.100 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 152 kB/s | 33 kB 00:00 2023-10-05T19:56:19.127 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): ceph-mgr-rook-17.2.6-1278.g487ffcb5.el8 194 kB/s | 47 kB 00:00 2023-10-05T19:56:19.224 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 605 kB/s | 81 kB 00:00 2023-10-05T19:56:19.296 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 848 kB/s | 61 kB 00:00 2023-10-05T19:56:19.324 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 961 kB/s | 27 kB 00:00 2023-10-05T19:56:19.337 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 9.5 MB/s | 126 kB 00:00 2023-10-05T19:56:19.350 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 8.3 MB/s | 110 kB 00:00 2023-10-05T19:56:19.363 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 1.6 MB/s | 20 kB 00:00 2023-10-05T19:56:19.376 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 12 MB/s | 155 kB 00:00 2023-10-05T19:56:19.382 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 252 kB/s | 64 kB 00:00 2023-10-05T19:56:19.387 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 3.6 MB/s | 43 kB 00:00 2023-10-05T19:56:19.398 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2023-10-05T19:56:19.401 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:19.401 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.2 MB/s | 1.9 MB 00:00 2023-10-05T19:56:19.407 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:19.421 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:19.421 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:19.507 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:19.511 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:20.009 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:20.238 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-05T19:56:20.491 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-05T19:56:20.761 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-05T19:56:21.057 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-10-05T19:56:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-10-05T19:56:21.501 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-10-05T19:56:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-10-05T19:56:21.975 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-05T19:56:22.168 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-05T19:56:22.413 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-05T19:56:23.564 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-05T19:56:23.869 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-05T19:56:24.008 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T19:56:24.307 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 13/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 1/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-10-05T19:56:24.308 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-10-05T19:56:24.309 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-05T19:56:24.309 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-10-05T19:56:24.309 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-10-05T19:56:25.400 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-10-05T19:56:25.401 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:29 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:27.336 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-05T19:56:27.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:27.436 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:27.580 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2023-10-05T19:56:27.954 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:28.788 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:32 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:30.631 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================================== 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================================== 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 121 k 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-05T19:56:30.632 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================================== 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout:Install 5 Packages 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:30.633 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.0 M 2023-10-05T19:56:30.634 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2023-10-05T19:56:30.634 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:30.934 INFO:teuthology.orchestra.run.smithi155.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1278.g487ffcb5.e 448 kB/s | 121 kB 00:00 2023-10-05T19:56:31.023 INFO:teuthology.orchestra.run.smithi155.stdout:(2/5): python3-babel-2.5.1-5.el8.noarch.rpm 54 MB/s | 4.8 MB 00:00 2023-10-05T19:56:31.043 INFO:teuthology.orchestra.run.smithi155.stdout:(3/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 25 MB/s | 538 kB 00:00 2023-10-05T19:56:31.091 INFO:teuthology.orchestra.run.smithi155.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 141 kB/s | 60 kB 00:00 2023-10-05T19:56:31.338 INFO:teuthology.orchestra.run.smithi155.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 704 kB/s | 476 kB 00:00 2023-10-05T19:56:31.342 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:31.342 INFO:teuthology.orchestra.run.smithi155.stdout:Total 8.4 MB/s | 6.0 MB 00:00 2023-10-05T19:56:31.345 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:31.355 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:31.356 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:31.462 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:31.467 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:32.499 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:32.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-10-05T19:56:33.144 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-10-05T19:56:33.447 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-05T19:56:33.678 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-05T19:56:33.832 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T19:56:34.127 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 5/5 2023-10-05T19:56:34.128 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 1/5 2023-10-05T19:56:34.128 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-05T19:56:34.128 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-05T19:56:34.128 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-10-05T19:56:35.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-10-05T19:56:35.152 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:39 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:37.079 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:37.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:37.178 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:37.296 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2023-10-05T19:56:37.671 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:38.509 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:42 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 876 k 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2023-10-05T19:56:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:40.374 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 958 k 2023-10-05T19:56:40.374 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 3.1 M 2023-10-05T19:56:40.374 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:40.430 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-10-05T19:56:40.801 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-17.2.6-1278.g487ffcb5.el8.x86_ 2.0 MB/s | 876 kB 00:00 2023-10-05T19:56:40.802 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:40.802 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.2 MB/s | 958 kB 00:00 2023-10-05T19:56:40.804 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:40.826 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:40.826 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:40.915 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:40.919 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:42.271 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:42.801 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-10-05T19:56:42.989 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T19:56:43.389 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2/2 2023-10-05T19:56:43.389 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/2 2023-10-05T19:56:44.449 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-10-05T19:56:44.449 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:48 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:46.361 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:46.458 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-10-05T19:56:46.458 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:46.458 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:46.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2023-10-05T19:56:46.957 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:47.782 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:51 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:49.577 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:49.578 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:49.578 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:56:49.578 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:17.2.6-1278.g487ffcb5.el8 ceph-noarch 267 k 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 267 k 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.3 M 2023-10-05T19:56:49.579 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:49.878 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-volume-17.2.6-1278.g487ffcb5.el8.noarch.rp 894 kB/s | 267 kB 00:00 2023-10-05T19:56:49.878 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:49.879 INFO:teuthology.orchestra.run.smithi155.stdout:Total 889 kB/s | 267 kB 00:00 2023-10-05T19:56:49.879 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:49.890 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:49.890 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:49.938 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:49.943 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:50.128 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:50.438 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:56:50.836 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:56:51.903 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 1/1 2023-10-05T19:56:51.903 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:55 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:53.830 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:56:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:53.927 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:56:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:17.2.6-1278.g487ffcb5.el8.noarch 2023-10-05T19:56:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:53.928 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:56:54.056 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2023-10-05T19:56:54.431 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:56:55.301 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:59 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:56:57.094 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:56:57.094 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 130 k 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:56:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 130 k 2023-10-05T19:56:57.096 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 464 k 2023-10-05T19:56:57.096 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:56:57.366 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-17.2.6-1278.g487ffcb5.el8.x86_64 480 kB/s | 130 kB 00:00 2023-10-05T19:56:57.366 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:56:57.366 INFO:teuthology.orchestra.run.smithi155.stdout:Total 478 kB/s | 130 kB 00:00 2023-10-05T19:56:57.369 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:56:57.374 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:56:57.374 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:56:57.429 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:56:57.433 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:56:57.552 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:56:57.744 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:56:58.024 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:56:59.398 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:56:59.398 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:03 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:01.243 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:57:01.343 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:01.343 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:57:01.343 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:57:01.343 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:01.343 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:01.469 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2023-10-05T19:57:01.846 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:02.717 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:06 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:04.198 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:04.677 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:04.678 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:04.678 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:04.786 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2023-10-05T19:57:05.159 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:06.038 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:10 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:07.817 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 25 k 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:07.818 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 25 k 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 109 k 2023-10-05T19:57:07.819 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:57:08.008 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-17.2.6-1278.g487ffcb5.el8.x86_6 131 kB/s | 25 kB 00:00 2023-10-05T19:57:08.008 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:57:08.009 INFO:teuthology.orchestra.run.smithi155.stdout:Total 130 kB/s | 25 kB 00:00 2023-10-05T19:57:08.009 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:57:08.013 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:57:08.013 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:57:08.014 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:57:08.019 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:57:08.070 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:57:08.250 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:09.447 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:09.448 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:13 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:11.290 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:57:11.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:11.387 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:57:11.387 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:57:11.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:11.387 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:11.526 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2023-10-05T19:57:11.900 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:12.764 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:16 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:14.088 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:14.560 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:14.561 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:14.561 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:14.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2023-10-05T19:57:15.028 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:15.883 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:19 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:17.213 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:17.685 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:17.686 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:17.686 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:17.789 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2023-10-05T19:57:18.159 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:19.060 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:23 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:20.420 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:20.895 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:20.897 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:20.897 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:21.006 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2023-10-05T19:57:21.384 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:22.235 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:26 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:23.620 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:24.099 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:24.100 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:24.100 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:24.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2023-10-05T19:57:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:25.474 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:29 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:26.819 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:27.288 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:27.289 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:27.289 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:27.397 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2023-10-05T19:57:27.770 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:28.615 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:32 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:30.154 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 is already installed. 2023-10-05T19:57:30.629 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:30.630 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2023-10-05T19:57:30.631 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:30.745 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2023-10-05T19:57:31.118 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:32.000 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:35 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:33.824 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:33.825 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:33.825 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2023-10-05T19:57:33.825 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 90 k 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 90 k 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 264 k 2023-10-05T19:57:33.826 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:57:34.085 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 348 kB/s | 90 kB 00:00 2023-10-05T19:57:34.086 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:57:34.086 INFO:teuthology.orchestra.run.smithi155.stdout:Total 346 kB/s | 90 kB 00:00 2023-10-05T19:57:34.087 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:57:34.098 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:57:34.098 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:57:34.153 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:57:34.157 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:57:34.271 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:57:34.479 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:34.736 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:35.819 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:35.819 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:39 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:37.678 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:57:37.776 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:37.776 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:57:37.776 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:57:37.776 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:37.776 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:37.890 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2023-10-05T19:57:38.264 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:39.155 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:43 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:40.971 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:40.971 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2023-10-05T19:57:40.971 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 3.5 M 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:57:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:40.973 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.5 M 2023-10-05T19:57:40.973 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 16 M 2023-10-05T19:57:40.973 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:57:41.291 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2023-10-05T19:57:41.293 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:57:41.293 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 3.5 MB 00:00 2023-10-05T19:57:41.293 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:57:41.306 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:57:41.306 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:57:41.377 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:57:41.381 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:57:41.827 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:57:41.996 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:42.414 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:43.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:43.483 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:47 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:45.318 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:57:45.417 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:45.417 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:57:45.417 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:57:45.417 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:45.417 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:45.536 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2023-10-05T19:57:45.914 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:57:46.776 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:50 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:48.608 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:57:48.609 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:48.609 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:17.2.6-1278.g487ffcb5.el8 ceph 173 k 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2023-10-05T19:57:48.610 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:48.611 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 173 k 2023-10-05T19:57:48.611 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 552 k 2023-10-05T19:57:48.611 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:57:48.891 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-17.2.6-1278.g487ffcb5.el8.x86_64.rpm 615 kB/s | 173 kB 00:00 2023-10-05T19:57:48.891 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:57:48.892 INFO:teuthology.orchestra.run.smithi155.stdout:Total 612 kB/s | 173 kB 00:00 2023-10-05T19:57:48.892 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:57:48.905 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:57:48.905 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:57:48.962 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:57:48.966 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:57:49.089 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:57:49.294 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:49.551 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 1/1 2023-10-05T19:57:50.592 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:54 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:57:52.457 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:57:52.555 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:52.555 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:57:52.555 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:17.2.6-1278.g487ffcb5.el8.x86_64 2023-10-05T19:57:52.555 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:52.555 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:57:52.697 DEBUG:teuthology.parallel:result is None 2023-10-05T19:57:52.697 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:57:52.915 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-05T19:57:52.969 INFO:teuthology.orchestra.run.smithi155.stdout:17.2.6-1278.g487ffcb5.el8 2023-10-05T19:57:52.969 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278.g487ffcb5.el8 2023-10-05T19:57:52.970 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278.g487ffcb5 is installed. 2023-10-05T19:57:52.970 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T19:57:52.973 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:52.973 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T19:57:53.034 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T19:57:53.035 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:53.035 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T19:57:53.098 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T19:57:53.161 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T19:57:53.162 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:53.162 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T19:57:53.226 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T19:57:53.290 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T19:57:53.290 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:53.291 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T19:57:53.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T19:57:53.419 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T19:57:53.483 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T19:57:53.483 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2023-10-05T19:57:53.507 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T19:57:53.507 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T19:57:53.571 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T19:57:53.571 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T19:57:53.643 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T19:57:53.643 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T19:57:53.643 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411769', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYiwqWwZ6dyEyEwETysEVt7X3DeM0X0sKB7EP5gkcGqxrvp1es/k5VfSSPvll5vzEXXKIz+V56EoZwhgmO4OoI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'transit', 'flavor': 'old', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T19:57:53.643 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T19:57:53.690 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T19:57:53.753 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:53.754 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T19:57:53.808 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T19:57:53.808 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2023-10-05T19:57:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T19:57:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:57:53.863 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 215931 Links: 1 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-10-05 19:57:42.180141644 +0000 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-10-05 19:53:14.416834708 +0000 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-10-05 19:53:14.416834708 +0000 2023-10-05T19:57:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-10-05T19:57:53.864 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T19:57:53.926 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-10-05T19:57:53.926 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-10-05T19:57:53.926 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.00024198 s, 2.1 MB/s 2023-10-05T19:57:53.927 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T19:57:53.983 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2023-10-05T19:57:54.038 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 217591 Links: 1 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-10-05 19:57:42.181141623 +0000 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-10-05 19:53:14.907824546 +0000 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-10-05 19:53:14.907824546 +0000 2023-10-05T19:57:54.039 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-10-05T19:57:54.040 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T19:57:54.101 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-10-05T19:57:54.101 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-10-05T19:57:54.101 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000227984 s, 2.2 MB/s 2023-10-05T19:57:54.102 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T19:57:54.158 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 217660 Links: 1 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:57:54.213 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-10-05 19:57:42.181141623 +0000 2023-10-05T19:57:54.214 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-10-05 19:53:15.400814343 +0000 2023-10-05T19:57:54.214 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-10-05 19:53:15.400814343 +0000 2023-10-05T19:57:54.214 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-10-05T19:57:54.214 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T19:57:54.276 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-10-05T19:57:54.276 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-10-05T19:57:54.276 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000243618 s, 2.1 MB/s 2023-10-05T19:57:54.277 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T19:57:54.333 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2023-10-05T19:57:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T19:57:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:57:54.388 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 220736 Links: 1 2023-10-05T19:57:54.388 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:57:54.388 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-10-05T19:57:54.389 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-10-05 19:57:42.181141623 +0000 2023-10-05T19:57:54.389 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-10-05 19:53:15.892804160 +0000 2023-10-05T19:57:54.389 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-10-05 19:53:15.892804160 +0000 2023-10-05T19:57:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-10-05T19:57:54.389 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T19:57:54.451 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-10-05T19:57:54.451 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-10-05T19:57:54.452 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000192204 s, 2.7 MB/s 2023-10-05T19:57:54.452 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T19:57:54.508 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:57:54.508 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:57:54.508 INFO:tasks.ceph:Generating config... 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/transit/export/encryption-key/ 2023-10-05T19:57:54.509 INFO:tasks.ceph:[client] rgw crypt vault secret engine = transit 2023-10-05T19:57:54.510 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T19:57:54.510 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T19:57:54.510 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T19:57:54.510 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T19:57:54.510 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T19:57:54.510 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T19:57:54.510 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T19:57:54.510 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T19:57:54.510 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T19:57:54.510 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-05T19:57:54.510 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T19:57:54.511 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T19:57:54.598 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T19:57:54.601 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T19:57:54.640 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:57:54.704 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.155')] 2023-10-05T19:57:54.704 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.155'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/export/encryption-key/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T19:57:54.705 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:54.705 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:57:54.759 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.155 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:57:54.834 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:57:54.834 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:57:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:57:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid 37cff2ea-3635-4882-b7b9-02c09cc5110d 2023-10-05T19:57:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:setting min_mon_release = octopus 2023-10-05T19:57:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2023-10-05T19:57:54.835 INFO:teuthology.orchestra.run.smithi155.stdout:fsid 37cff2ea-3635-4882-b7b9-02c09cc5110d 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2023-10-05T19:57:54.833961+0000 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:created 2023-10-05T19:57:54.833961+0000 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 15 (octopus) 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:election_strategy: 1 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2023-10-05T19:57:54.836 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T19:57:54.837 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:57:54.892 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 37cff2ea-3635-4882-b7b9-02c09cc5110d... 2023-10-05T19:57:54.893 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2023-10-05T19:57:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2023-10-05T19:57:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2023-10-05T19:57:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = 172.21.15.155 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = 37cff2ea-3635-4882-b7b9-02c09cc5110d 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2023-10-05T19:57:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2023-10-05T19:57:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2023-10-05T19:57:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2023-10-05T19:57:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2023-10-05T19:57:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T19:57:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2023-10-05T19:57:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: debug rgw = 20 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: rgw crypt require ssl = False 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: rgw crypt s3 kms backend = vault 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: rgw crypt vault auth = token 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: rgw crypt vault prefix = /v1/transit/export/encryption-key/ 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: rgw crypt vault secret engine = transit 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: setgroup = ceph 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: setuser = ceph 2023-10-05T19:57:54.986 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2023-10-05T19:57:54.987 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T19:57:54.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T19:57:55.078 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T19:57:55.078 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:55.079 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T19:57:55.093 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:55.093 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T19:57:55.147 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T19:57:55.148 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:55.148 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T19:57:55.148 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:57:55.222 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:57:55.222 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:57:55.277 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T19:57:55.277 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T19:57:55.277 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:57:55.365 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:57:55.367 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T19:57:55.367 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T19:57:55.368 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T19:57:55.405 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T19:57:55.417 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T19:57:55.418 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:57:55.418 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T19:57:55.481 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:57:55.481 INFO:tasks.ceph:role: osd.0 2023-10-05T19:57:55.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T19:57:55.482 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T19:57:55.553 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:57:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:57:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:57:55.553 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2023-10-05T19:57:55.553 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:57:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:57:55.554 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:57:55.554 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:57:55.554 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:57:55.554 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:57:55.983 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2023-10-05T19:57:55.985 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2023-10-05T19:57:55.985 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T19:57:56.173 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T19:57:56.204 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T19:57:56.269 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:57:56.269 INFO:tasks.ceph:role: osd.1 2023-10-05T19:57:56.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T19:57:56.269 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:57:56.341 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:57:56.342 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:57:56.765 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2023-10-05T19:57:56.766 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2023-10-05T19:57:56.766 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-05T19:57:56.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T19:57:56.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T19:57:56.960 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-05T19:57:56.960 INFO:tasks.ceph:role: osd.2 2023-10-05T19:57:56.960 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T19:57:56.960 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T19:57:57.032 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:57:57.032 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:57:57.032 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:57:57.032 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:57:57.033 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:57:57.485 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2023-10-05T19:57:57.487 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2023-10-05T19:57:57.487 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-05T19:57:57.647 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T19:57:57.677 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:57:57.766 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.764+0000 7fb53b3ac540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T19:57:57.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.767+0000 7fb53b3ac540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T19:57:57.770 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.767+0000 7fb53b3ac540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:57:57.770 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.767+0000 7fb53b3ac540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:57:57.770 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.767+0000 7fb53b3ac540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:57:57.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:57.770+0000 7fb53b3ac540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T19:57:58.297 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:58.294+0000 7fb53b3ac540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:57:59.060 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.057+0000 7fb53b3ac540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:57:59.853 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T19:57:59.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:57:59.965 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:57:59.962+0000 7fa4d65b9540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T19:58:00.488 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:00.486+0000 7fa4d65b9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:58:01.257 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:01.254+0000 7fa4d65b9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:58:02.048 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T19:58:02.117 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:58:02.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.198+0000 7f4b5af8f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T19:58:02.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.199+0000 7f4b5af8f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T19:58:02.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.199+0000 7f4b5af8f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:58:02.203 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.199+0000 7f4b5af8f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:58:02.203 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.199+0000 7f4b5af8f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:58:02.203 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.199+0000 7f4b5af8f540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T19:58:02.725 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:02.722+0000 7f4b5af8f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:58:03.490 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:03.487+0000 7f4b5af8f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:58:04.276 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T19:58:04.345 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T19:58:04.345 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:04.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T19:58:04.412 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:04.412 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T19:58:04.478 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:04.479 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T19:58:04.544 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:04.544 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T19:58:04.609 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:04.609 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T19:58:04.664 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T19:58:04.665 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T19:58:04.727 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBDFR9lJRK6FRAAckCnwrVjiAjclXFOeQH7mQ== 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBFFR9lPGWnLRAAulQwHaAbWr+Mh3NNLyB8Xw== 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBHFR9l6v9tORAA36borOOCK46S9Ghdy7DZgA== 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBKFR9lZv7vCxAAnU2M4xR/XO19d4cPE73Wdg== 2023-10-05T19:58:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2023-10-05T19:58:04.729 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBDFR9lLxkaGBAA0LsbmP59chmHGwQIFX6VmA== 2023-10-05T19:58:04.730 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T19:58:04.817 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:58:04.864 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:58:04.911 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:58:04.959 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T19:58:05.008 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T19:58:05.008 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T19:58:05.032 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T19:58:05.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T19:58:05.148 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:58:05.202 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T19:58:05.202 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T19:58:05.202 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T19:58:05.244 INFO:tasks.ceph.mon.a:Started 2023-10-05T19:58:05.245 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T19:58:05.245 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T19:58:05.245 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T19:58:05.247 INFO:tasks.ceph.mgr.x:Started 2023-10-05T19:58:05.247 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T19:58:05.247 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T19:58:05.448 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2023-10-05T19:58:05.459 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T19:58:05.459 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:05.459 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T19:58:05.483 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:05.484 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T19:58:05.548 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:58:05.549 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T19:58:05.613 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 07531411-8f81-469c-81ef-be6684f9f9d2 0 2023-10-05T19:58:05.862 INFO:teuthology.orchestra.run.smithi155.stdout:0 2023-10-05T19:58:05.874 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new e8640ac5-0586-4a42-9b90-0467c565203f 1 2023-10-05T19:58:06.080 INFO:teuthology.orchestra.run.smithi155.stdout:1 2023-10-05T19:58:06.092 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 1a698c1f-172f-4092-a286-167bef22592a 2 2023-10-05T19:58:06.304 INFO:teuthology.orchestra.run.smithi155.stdout:2 2023-10-05T19:58:06.316 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T19:58:06.317 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T19:58:06.318 INFO:tasks.ceph.osd.0:Started 2023-10-05T19:58:06.319 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T19:58:06.319 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T19:58:06.321 INFO:tasks.ceph.osd.1:Started 2023-10-05T19:58:06.321 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T19:58:06.321 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T19:58:06.323 INFO:tasks.ceph.osd.2:Started 2023-10-05T19:58:06.323 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:58:06.524 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:06.524 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":5,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T19:58:06.300900+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:58:06.536 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T19:58:06.536 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T19:58:06.734 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:06.732+0000 7fa77ef3c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T19:58:06.839 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:58:06.868 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:06.865+0000 7fa77ef3c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T19:58:06.926 INFO:tasks.ceph.osd.2.smithi155.stderr:2023-10-05T19:58:06.923+0000 7fb5905c6540 -1 Falling back to public interface 2023-10-05T19:58:06.935 INFO:tasks.ceph.osd.1.smithi155.stderr:2023-10-05T19:58:06.932+0000 7f718a651540 -1 Falling back to public interface 2023-10-05T19:58:06.937 INFO:tasks.ceph.osd.0.smithi155.stderr:2023-10-05T19:58:06.935+0000 7f7b5f1d6540 -1 Falling back to public interface 2023-10-05T19:58:06.948 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:06.945+0000 7fa77ef3c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T19:58:07.023 INFO:teuthology.misc.health.smithi155.stdout: 2023-10-05T19:58:07.023 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":5,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T19:58:06.300900+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:58:07.035 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-05T19:58:07.137 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:07.135+0000 7fa77ef3c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:58:07.312 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:07.309+0000 7fa77ef3c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T19:58:07.388 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:07.386+0000 7fa77ef3c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T19:58:07.534 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:07.531+0000 7fa77ef3c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T19:58:07.628 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:07.625+0000 7fa77ef3c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T19:58:08.137 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:08.135+0000 7fa77ef3c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T19:58:08.214 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:08.212+0000 7fa77ef3c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T19:58:08.291 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:08.288+0000 7fa77ef3c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T19:58:08.872 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:08.869+0000 7fa77ef3c180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-05T19:58:09.009 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:09.007+0000 7fa77ef3c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:58:09.101 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:09.098+0000 7fa77ef3c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T19:58:09.350 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:09.347+0000 7fa77ef3c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-05T19:58:09.431 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:09.428+0000 7fa77ef3c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T19:58:09.677 INFO:tasks.ceph.osd.1.smithi155.stderr:2023-10-05T19:58:09.675+0000 7f718a651540 -1 osd.1 0 log_to_monitors true 2023-10-05T19:58:09.681 INFO:tasks.ceph.osd.0.smithi155.stderr:2023-10-05T19:58:09.679+0000 7f7b5f1d6540 -1 osd.0 0 log_to_monitors true 2023-10-05T19:58:09.700 INFO:tasks.ceph.osd.2.smithi155.stderr:2023-10-05T19:58:09.697+0000 7fb5905c6540 -1 osd.2 0 log_to_monitors true 2023-10-05T19:58:10.258 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.256+0000 7fa77ef3c180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-05T19:58:10.331 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.328+0000 7fa77ef3c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T19:58:10.405 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.402+0000 7fa77ef3c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T19:58:10.538 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.535+0000 7fa77ef3c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T19:58:10.621 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.619+0000 7fa77ef3c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T19:58:10.789 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.786+0000 7fa77ef3c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T19:58:10.874 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.871+0000 7fa77ef3c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T19:58:10.958 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:10.956+0000 7fa77ef3c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T19:58:11.044 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:11.042+0000 7fa77ef3c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T19:58:11.919 INFO:tasks.ceph.osd.0.smithi155.stderr:2023-10-05T19:58:11.917+0000 7f7b55df7700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T19:58:11.921 INFO:tasks.ceph.osd.0.smithi155.stderr:2023-10-05T19:58:11.919+0000 7f7b4ebee700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:58:11.941 INFO:tasks.ceph.osd.2.smithi155.stderr:2023-10-05T19:58:11.939+0000 7fb5871e7700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T19:58:11.947 INFO:tasks.ceph.osd.2.smithi155.stderr:2023-10-05T19:58:11.944+0000 7fb57ffde700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:58:11.959 INFO:tasks.ceph.osd.1.smithi155.stderr:2023-10-05T19:58:11.957+0000 7f7181272700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T19:58:11.961 INFO:tasks.ceph.osd.1.smithi155.stderr:2023-10-05T19:58:11.959+0000 7f717a069700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:58:12.124 INFO:tasks.ceph.mgr.x.smithi155.stderr:2023-10-05T19:58:12.121+0000 7fa73833c700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T19:58:13.338 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:58:13.641 INFO:teuthology.misc.health.smithi155.stdout: 2023-10-05T19:58:13.641 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":8,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T19:58:12.919526+0000","last_up_change":"2023-10-05T19:58:12.919526+0000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":85992},{"type":"v1","addr":"172.21.15.155:6817","nonce":85992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":85992},{"type":"v1","addr":"172.21.15.155:6819","nonce":85992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":85992},{"type":"v1","addr":"172.21.15.155:6823","nonce":85992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":85992},{"type":"v1","addr":"172.21.15.155:6821","nonce":85992}]},"public_addr":"172.21.15.155:6817/85992","cluster_addr":"172.21.15.155:6819/85992","heartbeat_back_addr":"172.21.15.155:6823/85992","heartbeat_front_addr":"172.21.15.155:6821/85992","state":["exists","up"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":86001},{"type":"v1","addr":"172.21.15.155:6809","nonce":86001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":86001},{"type":"v1","addr":"172.21.15.155:6811","nonce":86001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":86001},{"type":"v1","addr":"172.21.15.155:6815","nonce":86001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":86001},{"type":"v1","addr":"172.21.15.155:6813","nonce":86001}]},"public_addr":"172.21.15.155:6809/86001","cluster_addr":"172.21.15.155:6811/86001","heartbeat_back_addr":"172.21.15.155:6815/86001","heartbeat_front_addr":"172.21.15.155:6813/86001","state":["exists","up"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":85985},{"type":"v1","addr":"172.21.15.155:6801","nonce":85985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":85985},{"type":"v1","addr":"172.21.15.155:6803","nonce":85985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":85985},{"type":"v1","addr":"172.21.15.155:6807","nonce":85985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":85985},{"type":"v1","addr":"172.21.15.155:6805","nonce":85985}]},"public_addr":"172.21.15.155:6801/85985","cluster_addr":"172.21.15.155:6803/85985","heartbeat_back_addr":"172.21.15.155:6807/85985","heartbeat_front_addr":"172.21.15.155:6805/85985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:58:13.653 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-05T19:58:13.653 INFO:tasks.ceph:Creating RBD pool 2023-10-05T19:58:13.654 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T19:58:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'rbd' created 2023-10-05T19:58:14.142 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T19:58:15.136 INFO:teuthology.orchestra.run.smithi155.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T19:58:15.150 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T19:58:15.150 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T19:58:15.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T19:58:15.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T19:58:15.151 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T19:58:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:15.571 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":85647},{"type":"v1","addr":"172.21.15.155:6825","nonce":85647}]},"active_addr":"172.21.15.155:6825/85647","active_change":"2023-10-05T19:58:11.056428+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":958776006}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":373712827}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":3735249406}]}]} 2023-10-05T19:58:15.573 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T19:58:15.574 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T19:58:15.574 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:58:15.884 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:15.885 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":10,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T19:58:15.124592+0000","last_up_change":"2023-10-05T19:58:12.919526+0000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:58:13.162201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:58:13.977145+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":85992},{"type":"v1","addr":"172.21.15.155:6817","nonce":85992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":85992},{"type":"v1","addr":"172.21.15.155:6819","nonce":85992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":85992},{"type":"v1","addr":"172.21.15.155:6823","nonce":85992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":85992},{"type":"v1","addr":"172.21.15.155:6821","nonce":85992}]},"public_addr":"172.21.15.155:6817/85992","cluster_addr":"172.21.15.155:6819/85992","heartbeat_back_addr":"172.21.15.155:6823/85992","heartbeat_front_addr":"172.21.15.155:6821/85992","state":["exists","up"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":86001},{"type":"v1","addr":"172.21.15.155:6809","nonce":86001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":86001},{"type":"v1","addr":"172.21.15.155:6811","nonce":86001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":86001},{"type":"v1","addr":"172.21.15.155:6815","nonce":86001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":86001},{"type":"v1","addr":"172.21.15.155:6813","nonce":86001}]},"public_addr":"172.21.15.155:6809/86001","cluster_addr":"172.21.15.155:6811/86001","heartbeat_back_addr":"172.21.15.155:6815/86001","heartbeat_front_addr":"172.21.15.155:6813/86001","state":["exists","up"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":85985},{"type":"v1","addr":"172.21.15.155:6801","nonce":85985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":85985},{"type":"v1","addr":"172.21.15.155:6803","nonce":85985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":85985},{"type":"v1","addr":"172.21.15.155:6807","nonce":85985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":85985},{"type":"v1","addr":"172.21.15.155:6805","nonce":85985}]},"public_addr":"172.21.15.155:6801/85985","cluster_addr":"172.21.15.155:6803/85985","heartbeat_back_addr":"172.21.15.155:6807/85985","heartbeat_front_addr":"172.21.15.155:6805/85985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.670100+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.645307+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.703975+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:58:15.898 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T19:58:15.899 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:58:16.207 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:16.207 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":11,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T19:58:16.126166+0000","last_up_change":"2023-10-05T19:58:12.919526+0000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:58:13.162201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:58:13.977145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":85992},{"type":"v1","addr":"172.21.15.155:6817","nonce":85992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":85992},{"type":"v1","addr":"172.21.15.155:6819","nonce":85992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":85992},{"type":"v1","addr":"172.21.15.155:6823","nonce":85992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":85992},{"type":"v1","addr":"172.21.15.155:6821","nonce":85992}]},"public_addr":"172.21.15.155:6817/85992","cluster_addr":"172.21.15.155:6819/85992","heartbeat_back_addr":"172.21.15.155:6823/85992","heartbeat_front_addr":"172.21.15.155:6821/85992","state":["exists","up"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":86001},{"type":"v1","addr":"172.21.15.155:6809","nonce":86001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":86001},{"type":"v1","addr":"172.21.15.155:6811","nonce":86001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":86001},{"type":"v1","addr":"172.21.15.155:6815","nonce":86001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":86001},{"type":"v1","addr":"172.21.15.155:6813","nonce":86001}]},"public_addr":"172.21.15.155:6809/86001","cluster_addr":"172.21.15.155:6811/86001","heartbeat_back_addr":"172.21.15.155:6815/86001","heartbeat_front_addr":"172.21.15.155:6813/86001","state":["exists","up"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":85985},{"type":"v1","addr":"172.21.15.155:6801","nonce":85985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":85985},{"type":"v1","addr":"172.21.15.155:6803","nonce":85985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":85985},{"type":"v1","addr":"172.21.15.155:6807","nonce":85985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":85985},{"type":"v1","addr":"172.21.15.155:6805","nonce":85985}]},"public_addr":"172.21.15.155:6801/85985","cluster_addr":"172.21.15.155:6803/85985","heartbeat_back_addr":"172.21.15.155:6807/85985","heartbeat_front_addr":"172.21.15.155:6805/85985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.670100+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.645307+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.703975+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:58:16.221 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T19:58:16.221 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T19:58:16.222 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T19:58:16.379 INFO:teuthology.orchestra.run.smithi155.stdout:34359738370 2023-10-05T19:58:16.380 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:58:16.387 INFO:teuthology.orchestra.run.smithi155.stdout:34359738370 2023-10-05T19:58:16.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:58:16.388 INFO:teuthology.orchestra.run.smithi155.stdout:34359738371 2023-10-05T19:58:16.388 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:58:16.715 INFO:teuthology.orchestra.run.smithi155.stdout:34359738369 2023-10-05T19:58:16.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2023-10-05T19:58:16.754 INFO:teuthology.orchestra.run.smithi155.stdout:34359738369 2023-10-05T19:58:16.764 INFO:teuthology.orchestra.run.smithi155.stdout:34359738370 2023-10-05T19:58:16.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.1 2023-10-05T19:58:16.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.2 2023-10-05T19:58:17.730 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:58:17.769 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:58:17.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:58:18.077 INFO:teuthology.orchestra.run.smithi155.stdout:34359738370 2023-10-05T19:58:18.091 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.0 2023-10-05T19:58:18.091 DEBUG:teuthology.parallel:result is None 2023-10-05T19:58:18.101 INFO:teuthology.orchestra.run.smithi155.stdout:34359738370 2023-10-05T19:58:18.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2023-10-05T19:58:18.116 DEBUG:teuthology.parallel:result is None 2023-10-05T19:58:18.153 INFO:teuthology.orchestra.run.smithi155.stdout:34359738371 2023-10-05T19:58:18.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.2 2023-10-05T19:58:18.167 DEBUG:teuthology.parallel:result is None 2023-10-05T19:58:18.167 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T19:58:18.167 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:58:18.508 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:18.510 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2023-10-05T19:58:18.521 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-05T19:58:17.122536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893416,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679384,"statfs":{"total":322122547200,"available":321207689216,"internally_reserved":0,"allocated":1458176,"data_stored":1248104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138511+0000","last_change":"2023-10-05T19:58:15.136664+0000","last_active":"2023-10-05T19:58:15.138511+0000","last_peered":"2023-10-05T19:58:15.138511+0000","last_clean":"2023-10-05T19:58:15.138511+0000","last_became_active":"2023-10-05T19:58:15.136415+0000","last_became_peered":"2023-10-05T19:58:15.136415+0000","last_unstale":"2023-10-05T19:58:15.138511+0000","last_undegraded":"2023-10-05T19:58:15.138511+0000","last_fullsized":"2023-10-05T19:58:15.138511+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:13:08.921006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138545+0000","last_change":"2023-10-05T19:58:15.136693+0000","last_active":"2023-10-05T19:58:15.138545+0000","last_peered":"2023-10-05T19:58:15.138545+0000","last_clean":"2023-10-05T19:58:15.138545+0000","last_became_active":"2023-10-05T19:58:15.136472+0000","last_became_peered":"2023-10-05T19:58:15.136472+0000","last_unstale":"2023-10-05T19:58:15.138545+0000","last_undegraded":"2023-10-05T19:58:15.138545+0000","last_fullsized":"2023-10-05T19:58:15.138545+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:47:13.833180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138550+0000","last_change":"2023-10-05T19:58:15.136867+0000","last_active":"2023-10-05T19:58:15.138550+0000","last_peered":"2023-10-05T19:58:15.138550+0000","last_clean":"2023-10-05T19:58:15.138550+0000","last_became_active":"2023-10-05T19:58:15.136588+0000","last_became_peered":"2023-10-05T19:58:15.136588+0000","last_unstale":"2023-10-05T19:58:15.138550+0000","last_undegraded":"2023-10-05T19:58:15.138550+0000","last_fullsized":"2023-10-05T19:58:15.138550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:39:57.929484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133283+0000","last_change":"2023-10-05T19:58:15.133283+0000","last_active":"2023-10-05T19:58:15.133283+0000","last_peered":"2023-10-05T19:58:15.133283+0000","last_clean":"2023-10-05T19:58:15.133283+0000","last_became_active":"2023-10-05T19:58:15.133072+0000","last_became_peered":"2023-10-05T19:58:15.133072+0000","last_unstale":"2023-10-05T19:58:15.133283+0000","last_undegraded":"2023-10-05T19:58:15.133283+0000","last_fullsized":"2023-10-05T19:58:15.133283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:43:21.173998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.575699+0000","last_change":"2023-10-05T19:58:15.135820+0000","last_active":"2023-10-05T19:58:15.575699+0000","last_peered":"2023-10-05T19:58:15.575699+0000","last_clean":"2023-10-05T19:58:15.575699+0000","last_became_active":"2023-10-05T19:58:15.134947+0000","last_became_peered":"2023-10-05T19:58:15.134947+0000","last_unstale":"2023-10-05T19:58:15.575699+0000","last_undegraded":"2023-10-05T19:58:15.575699+0000","last_fullsized":"2023-10-05T19:58:15.575699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:31:12.925705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133207+0000","last_change":"2023-10-05T19:58:15.131818+0000","last_active":"2023-10-05T19:58:15.133207+0000","last_peered":"2023-10-05T19:58:15.133207+0000","last_clean":"2023-10-05T19:58:15.133207+0000","last_became_active":"2023-10-05T19:58:15.131541+0000","last_became_peered":"2023-10-05T19:58:15.131541+0000","last_unstale":"2023-10-05T19:58:15.133207+0000","last_undegraded":"2023-10-05T19:58:15.133207+0000","last_fullsized":"2023-10-05T19:58:15.133207+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:20:01.724713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.132934+0000","last_change":"2023-10-05T19:58:15.132934+0000","last_active":"2023-10-05T19:58:15.132934+0000","last_peered":"2023-10-05T19:58:15.132934+0000","last_clean":"2023-10-05T19:58:15.132934+0000","last_became_active":"2023-10-05T19:58:15.132687+0000","last_became_peered":"2023-10-05T19:58:15.132687+0000","last_unstale":"2023-10-05T19:58:15.132934+0000","last_undegraded":"2023-10-05T19:58:15.132934+0000","last_fullsized":"2023-10-05T19:58:15.132934+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:21:06.595451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133063+0000","last_change":"2023-10-05T19:58:15.133063+0000","last_active":"2023-10-05T19:58:15.133063+0000","last_peered":"2023-10-05T19:58:15.133063+0000","last_clean":"2023-10-05T19:58:15.133063+0000","last_became_active":"2023-10-05T19:58:15.132843+0000","last_became_peered":"2023-10-05T19:58:15.132843+0000","last_unstale":"2023-10-05T19:58:15.133063+0000","last_undegraded":"2023-10-05T19:58:15.133063+0000","last_fullsized":"2023-10-05T19:58:15.133063+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:07.308401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138515+0000","last_change":"2023-10-05T19:58:15.136672+0000","last_active":"2023-10-05T19:58:15.138515+0000","last_peered":"2023-10-05T19:58:15.138515+0000","last_clean":"2023-10-05T19:58:15.138515+0000","last_became_active":"2023-10-05T19:58:15.136408+0000","last_became_peered":"2023-10-05T19:58:15.136408+0000","last_unstale":"2023-10-05T19:58:15.138515+0000","last_undegraded":"2023-10-05T19:58:15.138515+0000","last_fullsized":"2023-10-05T19:58:15.138515+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:30:36.802224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297376,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560224,"statfs":{"total":107374182400,"available":107069669376,"internally_reserved":0,"allocated":90112,"data_stored":22456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297956,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559644,"statfs":{"total":107374182400,"available":107069075456,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298084,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559516,"statfs":{"total":107374182400,"available":107068944384,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:58:18.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:58:18.825 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2023-10-05T19:58:18.838 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-05T19:58:17.122536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893416,"kb_used_data":1424,"kb_used_omap":0,"kb_used_meta":891840,"kb_avail":313679384,"statfs":{"total":322122547200,"available":321207689216,"internally_reserved":0,"allocated":1458176,"data_stored":1248104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":913244160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138511+0000","last_change":"2023-10-05T19:58:15.136664+0000","last_active":"2023-10-05T19:58:15.138511+0000","last_peered":"2023-10-05T19:58:15.138511+0000","last_clean":"2023-10-05T19:58:15.138511+0000","last_became_active":"2023-10-05T19:58:15.136415+0000","last_became_peered":"2023-10-05T19:58:15.136415+0000","last_unstale":"2023-10-05T19:58:15.138511+0000","last_undegraded":"2023-10-05T19:58:15.138511+0000","last_fullsized":"2023-10-05T19:58:15.138511+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:13:08.921006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138545+0000","last_change":"2023-10-05T19:58:15.136693+0000","last_active":"2023-10-05T19:58:15.138545+0000","last_peered":"2023-10-05T19:58:15.138545+0000","last_clean":"2023-10-05T19:58:15.138545+0000","last_became_active":"2023-10-05T19:58:15.136472+0000","last_became_peered":"2023-10-05T19:58:15.136472+0000","last_unstale":"2023-10-05T19:58:15.138545+0000","last_undegraded":"2023-10-05T19:58:15.138545+0000","last_fullsized":"2023-10-05T19:58:15.138545+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:47:13.833180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138550+0000","last_change":"2023-10-05T19:58:15.136867+0000","last_active":"2023-10-05T19:58:15.138550+0000","last_peered":"2023-10-05T19:58:15.138550+0000","last_clean":"2023-10-05T19:58:15.138550+0000","last_became_active":"2023-10-05T19:58:15.136588+0000","last_became_peered":"2023-10-05T19:58:15.136588+0000","last_unstale":"2023-10-05T19:58:15.138550+0000","last_undegraded":"2023-10-05T19:58:15.138550+0000","last_fullsized":"2023-10-05T19:58:15.138550+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:39:57.929484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133283+0000","last_change":"2023-10-05T19:58:15.133283+0000","last_active":"2023-10-05T19:58:15.133283+0000","last_peered":"2023-10-05T19:58:15.133283+0000","last_clean":"2023-10-05T19:58:15.133283+0000","last_became_active":"2023-10-05T19:58:15.133072+0000","last_became_peered":"2023-10-05T19:58:15.133072+0000","last_unstale":"2023-10-05T19:58:15.133283+0000","last_undegraded":"2023-10-05T19:58:15.133283+0000","last_fullsized":"2023-10-05T19:58:15.133283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:43:21.173998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.575699+0000","last_change":"2023-10-05T19:58:15.135820+0000","last_active":"2023-10-05T19:58:15.575699+0000","last_peered":"2023-10-05T19:58:15.575699+0000","last_clean":"2023-10-05T19:58:15.575699+0000","last_became_active":"2023-10-05T19:58:15.134947+0000","last_became_peered":"2023-10-05T19:58:15.134947+0000","last_unstale":"2023-10-05T19:58:15.575699+0000","last_undegraded":"2023-10-05T19:58:15.575699+0000","last_fullsized":"2023-10-05T19:58:15.575699+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:31:12.925705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133207+0000","last_change":"2023-10-05T19:58:15.131818+0000","last_active":"2023-10-05T19:58:15.133207+0000","last_peered":"2023-10-05T19:58:15.133207+0000","last_clean":"2023-10-05T19:58:15.133207+0000","last_became_active":"2023-10-05T19:58:15.131541+0000","last_became_peered":"2023-10-05T19:58:15.131541+0000","last_unstale":"2023-10-05T19:58:15.133207+0000","last_undegraded":"2023-10-05T19:58:15.133207+0000","last_fullsized":"2023-10-05T19:58:15.133207+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T21:20:01.724713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.132934+0000","last_change":"2023-10-05T19:58:15.132934+0000","last_active":"2023-10-05T19:58:15.132934+0000","last_peered":"2023-10-05T19:58:15.132934+0000","last_clean":"2023-10-05T19:58:15.132934+0000","last_became_active":"2023-10-05T19:58:15.132687+0000","last_became_peered":"2023-10-05T19:58:15.132687+0000","last_unstale":"2023-10-05T19:58:15.132934+0000","last_undegraded":"2023-10-05T19:58:15.132934+0000","last_fullsized":"2023-10-05T19:58:15.132934+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T23:21:06.595451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.133063+0000","last_change":"2023-10-05T19:58:15.133063+0000","last_active":"2023-10-05T19:58:15.133063+0000","last_peered":"2023-10-05T19:58:15.133063+0000","last_clean":"2023-10-05T19:58:15.133063+0000","last_became_active":"2023-10-05T19:58:15.132843+0000","last_became_peered":"2023-10-05T19:58:15.132843+0000","last_unstale":"2023-10-05T19:58:15.133063+0000","last_undegraded":"2023-10-05T19:58:15.133063+0000","last_fullsized":"2023-10-05T19:58:15.133063+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:07.308401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-10-05T19:58:15.138515+0000","last_change":"2023-10-05T19:58:15.136672+0000","last_active":"2023-10-05T19:58:15.138515+0000","last_peered":"2023-10-05T19:58:15.138515+0000","last_clean":"2023-10-05T19:58:15.138515+0000","last_became_active":"2023-10-05T19:58:15.136408+0000","last_became_peered":"2023-10-05T19:58:15.136408+0000","last_unstale":"2023-10-05T19:58:15.138515+0000","last_undegraded":"2023-10-05T19:58:15.138515+0000","last_fullsized":"2023-10-05T19:58:15.138515+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:58:14.123207+0000","last_clean_scrub_stamp":"2023-10-05T19:58:14.123207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:30:36.802224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297376,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104560224,"statfs":{"total":107374182400,"available":107069669376,"internally_reserved":0,"allocated":90112,"data_stored":22456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297956,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559644,"statfs":{"total":107374182400,"available":107069075456,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298084,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":104559516,"statfs":{"total":107374182400,"available":107068944384,"internally_reserved":0,"allocated":684032,"data_stored":612824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:58:18.839 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T19:58:18.840 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T19:58:18.840 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T19:58:18.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T19:58:19.185 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:19.185 INFO:teuthology.orchestra.run.smithi155.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T19:58:19.199 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T19:58:19.199 INFO:teuthology.run_tasks:Running task vault... 2023-10-05T19:58:19.211 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'transit', 'flavor': 'old', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}} 2023-10-05T19:58:19.212 INFO:tasks.vault:Downloading Vault... 2023-10-05T19:58:19.212 INFO:tasks.vault:Downloading Vault... 2023-10-05T19:58:19.212 DEBUG:teuthology.orchestra.run.smithi155:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2023-10-05T19:58:19.239 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:19.239 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:19.983 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 24 42.6M 24 10.4M 0 0 45.8M 0 --:--:-- --:--:-- --:--:-- 45.6M 100 42.6M 100 42.6M 0 0 57.2M 0 --:--:-- --:--:-- --:--:-- 57.2M 2023-10-05T19:58:19.985 INFO:tasks.vault:Verifying SHA256 signature... 2023-10-05T19:58:19.985 DEBUG:teuthology.orchestra.run.smithi155:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2023-10-05T19:58:20.149 INFO:tasks.vault:Extracting vault... 2023-10-05T19:58:20.150 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p /home/ubuntu/cephtest/vault 2023-10-05T19:58:20.164 DEBUG:teuthology.orchestra.run.smithi155:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2023-10-05T19:58:20.236 INFO:teuthology.orchestra.run.smithi155.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2023-10-05T19:58:20.958 INFO:teuthology.orchestra.run.smithi155.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2023-10-05T19:58:20.959 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2023-10-05T19:58:20.959 INFO:tasks.vault.client.0:Restarting daemon 2023-10-05T19:58:20.959 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2023-10-05T19:58:21.001 INFO:tasks.vault.client.0:Started 2023-10-05T19:58:21.042 INFO:tasks.vault.client.0.smithi155.stdout:==> Vault server configuration: 2023-10-05T19:58:21.043 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.043 INFO:tasks.vault.client.0.smithi155.stdout: Api Address: http://0.0.0.0:8200 2023-10-05T19:58:21.043 INFO:tasks.vault.client.0.smithi155.stdout: Cgo: disabled 2023-10-05T19:58:21.043 INFO:tasks.vault.client.0.smithi155.stdout: Cluster Address: https://0.0.0.0:8201 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: Log Level: info 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: Mlock: supported: true, enabled: false 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: Storage: inmem 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: Version: Vault v1.2.2 2023-10-05T19:58:21.047 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout:and starts unsealed with a single unseal key. The root token is already 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout:You may need to set the following environment variable: 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.061 INFO:tasks.vault.client.0.smithi155.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:The unseal key and root token are displayed below in case you want to 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:seal/unseal the Vault or re-authenticate. 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:Unseal Key: AvGk2O775aSJnY0oRdbqtMuk32PGIde2j6m0uFPrwOA= 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:Root Token: test_root_token 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:Development mode should NOT be used in production installations! 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout:==> Vault server started! Log data will stream in below: 2023-10-05T19:58:21.062 INFO:tasks.vault.client.0.smithi155.stdout: 2023-10-05T19:58:21.063 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.039Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2023-10-05T19:58:21.063 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.040Z [INFO] core: security barrier not initialized 2023-10-05T19:58:21.063 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.040Z [INFO] core: security barrier initialized: shares=1 threshold=1 2023-10-05T19:58:21.063 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.041Z [INFO] core: post-unseal setup starting 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.049Z [INFO] core: loaded wrapping token key 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.049Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.049Z [INFO] core: no mounts; adding default mount table 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.049Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.050Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.050Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.050Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] core: restoring leases 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] rollback: starting rollback manager 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] expiration: lease restore complete 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] identity: entities restored 2023-10-05T19:58:21.064 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] identity: groups restored 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] core: post-unseal setup complete 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] core: root token generated 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] core: pre-seal teardown starting 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.051Z [INFO] rollback: stopping rollback manager 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: pre-seal teardown complete 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: post-unseal setup starting 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: loaded wrapping token key 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2023-10-05T19:58:21.065 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: successfully mounted backend: type=system path=sys/ 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.052Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] core: restoring leases 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] rollback: starting rollback manager 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] identity: entities restored 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] identity: groups restored 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] core: post-unseal setup complete 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] core: vault is unsealed 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.053Z [INFO] expiration: lease restore complete 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.054Z [INFO] expiration: revoked lease: lease_id=auth/token/root/h4212fc1c95daac9fb695f3907a1a66d937ff9624f4a0ac1cbb313827b262adbd 2023-10-05T19:58:21.066 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.055Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2023-10-05T19:58:21.067 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.065Z [INFO] secrets.kv.kv_0ec1bc39: collecting keys to upgrade 2023-10-05T19:58:21.067 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.065Z [INFO] secrets.kv.kv_0ec1bc39: done collecting keys: num_keys=1 2023-10-05T19:58:21.067 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:21.065Z [INFO] secrets.kv.kv_0ec1bc39: upgrading keys finished 2023-10-05T19:58:31.001 INFO:tasks.vault:Mounting transit secrets engine 2023-10-05T19:58:31.001 INFO:tasks.vault:Send request to Vault: smithi155.front.sepia.ceph.com:8200 at /v1/sys/mounts/transit with token: test_root_token 2023-10-05T19:58:31.026 INFO:tasks.vault:b'' 2023-10-05T19:58:31.026 INFO:tasks.vault:Send request to Vault: smithi155.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-1 with token: test_root_token 2023-10-05T19:58:31.027 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T19:58:31.025Z [INFO] core: successful mount: namespace= path=transit/ type=transit 2023-10-05T19:58:31.029 INFO:tasks.vault:b'' 2023-10-05T19:58:31.029 INFO:tasks.vault:Send request to Vault: smithi155.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-2 with token: test_root_token 2023-10-05T19:58:31.031 INFO:tasks.vault:b'' 2023-10-05T19:58:31.031 INFO:tasks.vault:secrets created 2023-10-05T19:58:31.032 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T19:58:31.045 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2023-10-05T19:58:31.045 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T19:58:31.045 INFO:tasks.rgw:Creating data pools 2023-10-05T19:58:31.045 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T19:58:31.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-10-05T19:58:32.352 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T19:58:32.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T19:58:33.358 INFO:teuthology.orchestra.run.smithi155.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T19:58:33.373 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T19:58:34.359 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T19:58:34.373 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T19:58:35.372 INFO:teuthology.orchestra.run.smithi155.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T19:58:35.386 DEBUG:tasks.rgw:Pools created 2023-10-05T19:58:35.387 INFO:tasks.rgw:Starting rgw... 2023-10-05T19:58:35.387 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2023-10-05T19:58:35.387 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T19:58:35.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2023-10-05T19:58:35.412 INFO:teuthology.orchestra.run.smithi155.stdout:test_root_token 2023-10-05T19:58:35.413 INFO:tasks.rgw:Token file content 2023-10-05T19:58:35.413 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/ceph/vault-root-token 2023-10-05T19:58:35.466 INFO:teuthology.orchestra.run.smithi155.stdout:test_root_token 2023-10-05T19:58:35.467 INFO:tasks.rgw:Restrict access to token file 2023-10-05T19:58:35.467 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 600 /etc/ceph/vault-root-token 2023-10-05T19:58:35.530 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown ceph /etc/ceph/vault-root-token 2023-10-05T19:58:35.594 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T19:58:35.595 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi155.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi155.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T19:58:35.637 INFO:tasks.rgw.client.0:Started 2023-10-05T19:58:35.637 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:35.637 DEBUG:teuthology.orchestra.run.smithi155:> curl http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:58:35.655 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:35.655 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:35.655 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi155.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:58:36.657 DEBUG:teuthology.orchestra.run.smithi155:> curl http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:36.676 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:36.676 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:36.676 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi155.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:58:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:58:38.678 DEBUG:teuthology.orchestra.run.smithi155:> curl http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:38.700 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:38.700 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:58:38.701 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi155.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:58:42.702 DEBUG:teuthology.orchestra.run.smithi155:> curl http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:42.724 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:42.724 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:58:42.726 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi155.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:58:50.727 DEBUG:teuthology.orchestra.run.smithi155:> curl http://smithi155.front.sepia.ceph.com:80/ 2023-10-05T19:58:50.749 INFO:teuthology.orchestra.run.smithi155.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:58:50.749 INFO:teuthology.orchestra.run.smithi155.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:58:50.750 INFO:teuthology.orchestra.run.smithi155.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-10-05T19:58:50.751 INFO:teuthology.orchestra.run.smithi155.stdout:anonymous 2023-10-05T19:58:50.751 INFO:teuthology.run_tasks:Running task s3tests... 2023-10-05T19:58:50.766 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-10-05T19:58:50.766 INFO:tasks.s3tests:Downloading s3-tests... 2023-10-05T19:58:50.766 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-10-05T19:58:50.767 DEBUG:teuthology.orchestra.run.smithi155:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:58:50.808 INFO:teuthology.orchestra.run.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-client.0'... 2023-10-05T19:58:51.668 INFO:tasks.s3tests:Creating rgw users... 2023-10-05T19:58:51.668 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-10-05T19:58:51.669 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key KGAWZTWMIAQUYPOGNDEV --secret MYGFqzlXRD9uAR9Eiwd6vztC2c7xG9fwOs/fxZ8CQDepCv/KxJ6pUg== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:58:51.757 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:51.757 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:51.762 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.760+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 realm 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.764 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.761+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.762+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:51.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.762+0000 7fd1a506f680 20 searching for the correct realm 2023-10-05T19:58:51.772 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:51.772 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:51.772 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.769+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.770+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.770+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.770+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.773 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.770+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.774 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.770+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.779 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.780 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.780 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:51.780 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.776+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.780 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.780 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.781 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:51.781 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.781 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 zone default found 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 4 Realm: () 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 zonegroup default 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.777+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.782 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.778+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.783 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.778+0000 7fd1a506f680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.783 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.778+0000 7fd1a506f680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.783 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.778+0000 7fd1a506f680 20 started sync module instance, tier type = 2023-10-05T19:58:51.783 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.778+0000 7fd1a506f680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=0 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=1 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=2 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=3 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=4 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=5 2023-10-05T19:58:51.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.799+0000 7fd1a506f680 20 add_watcher() i=6 2023-10-05T19:58:51.803 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.800+0000 7fd1a506f680 20 add_watcher() i=7 2023-10-05T19:58:51.803 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.800+0000 7fd1a506f680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:51.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.802+0000 7fd169fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:51.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.802+0000 7fd1a506f680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:51.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.802+0000 7fd1a506f680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:51.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.806+0000 7fd1a506f680 5 note: GC not initialized 2023-10-05T19:58:51.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.807+0000 7fd0e0ff9700 20 reqs_thread_entry: start 2023-10-05T19:58:51.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.807+0000 7fd0d67fc700 20 reqs_thread_entry: start 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.807+0000 7fd1a506f680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.807+0000 7fd1a506f680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd0d4ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:22 2023 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd0bbfff700 20 reqs_thread_entry: start 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 20 get_system_obj_state: rctx=0x7fffc20219b8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557abba67240 s->prefetch_data=0 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 20 get_system_obj_state: rctx=0x7fffc20216a8 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x557abba67240 s->prefetch_data=0 2023-10-05T19:58:51.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 20 get_system_obj_state: rctx=0x7fffc20216a8 obj=default.rgw.meta:users.keys:KGAWZTWMIAQUYPOGNDEV state=0x557abba67240 s->prefetch_data=0 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache get: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV : miss 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV info.flags=0x0 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.808+0000 7fd1a506f680 10 adding default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV to cache LRU end 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.809+0000 7fd1a506f680 20 get_system_obj_state: rctx=0x7fffc2020f38 obj=default.rgw.meta:users.keys:KGAWZTWMIAQUYPOGNDEV state=0x557abba67240 s->prefetch_data=0 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.809+0000 7fd1a506f680 10 cache get: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV : hit (negative entry) 2023-10-05T19:58:51.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.809+0000 7fd1a506f680 20 get_system_obj_state: rctx=0x7fffc20215e8 obj=default.rgw.meta:users.keys:KGAWZTWMIAQUYPOGNDEV state=0x557abba67240 s->prefetch_data=0 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.809+0000 7fd1a506f680 10 cache get: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV : hit (negative entry) 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd1a506f680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd1a506f680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd196164700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215104 cookie 93985624245104 notifier 4211 bl.length()=640 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd196164700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:58:51.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.810+0000 7fd196164700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd1a506f680 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd1a506f680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd195963700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215104 cookie 93985624259072 notifier 4211 bl.length()=194 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd195963700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:51.815 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.812+0000 7fd195963700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-10-05T19:58:51.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd1a506f680 10 cache put: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV info.flags=0x7 2023-10-05T19:58:51.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd1a506f680 10 moving default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV to cache LRU end 2023-10-05T19:58:51.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd1a506f680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:KGAWZTWMIAQUYPOGNDEV, ofs0, ns] 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd196164700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 103079215105 cookie 93985624245104 notifier 4211 bl.length()=186 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd196164700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV info.flags=0x7 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.815+0000 7fd196164700 10 rgw watcher librados: moving default.rgw.meta+users.keys+KGAWZTWMIAQUYPOGNDEV to cache LRU end 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stdout: "user_id": "foo.client.0", 2023-10-05T19:58:51.818 INFO:teuthology.orchestra.run.smithi155.stdout: "display_name": "Mr. foo.client.0", 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "email": "foo.client.0+test@test.test", 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "suspended": 0, 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "max_buckets": 1000, 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "subusers": [], 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "keys": [ 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "user": "foo.client.0", 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "access_key": "KGAWZTWMIAQUYPOGNDEV", 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "secret_key": "MYGFqzlXRD9uAR9Eiwd6vztC2c7xG9fwOs/fxZ8CQDepCv/KxJ6pUg==" 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:51.819 INFO:teuthology.orchestra.run.smithi155.stdout: "swift_keys": [], 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": [ 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "user-policy", 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "perm": "*" 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "op_mask": "read, write, delete", 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "default_placement": "", 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "default_storage_class": "", 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "placement_tags": [], 2023-10-05T19:58:51.820 INFO:teuthology.orchestra.run.smithi155.stdout: "bucket_quota": { 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "user_quota": { 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:51.821 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout: "temp_url_keys": [], 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "rgw", 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout: "mfa_ids": [] 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-10-05T19:58:51.822 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:51.823 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.816+0000 7fd1a506f680 20 remove_watcher() i=0 2023-10-05T19:58:51.823 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.816+0000 7fd1a506f680 2 removed watcher, disabling cache 2023-10-05T19:58:51.823 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.817+0000 7fd1a506f680 20 remove_watcher() i=1 2023-10-05T19:58:51.823 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.820+0000 7fd1a506f680 20 remove_watcher() i=2 2023-10-05T19:58:51.823 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.821+0000 7fd1a506f680 20 remove_watcher() i=3 2023-10-05T19:58:51.824 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.821+0000 7fd1a506f680 20 remove_watcher() i=4 2023-10-05T19:58:51.825 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.822+0000 7fd1a506f680 20 remove_watcher() i=5 2023-10-05T19:58:51.826 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.824+0000 7fd1a506f680 20 remove_watcher() i=6 2023-10-05T19:58:51.827 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.825+0000 7fd1a506f680 20 remove_watcher() i=7 2023-10-05T19:58:51.827 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.825+0000 7fd169fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:51.832 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 3895401634 --totp-seed O6UOVYRSGRKJVMAM3DT2AGFAJ7QVCOX3UCQDCHXM4Q7XXZPHEU4SZSU2L3SHE7V7 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:58:51.880 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:51.880 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:51.885 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.883+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.887 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.884+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.887 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.884+0000 7f3199f94680 20 realm 2023-10-05T19:58:51.887 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.884+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.884+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.884+0000 7f3199f94680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.885+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.885+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.885+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.885+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.888 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.885+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.886+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:51.889 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.886+0000 7f3199f94680 20 searching for the correct realm 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.892+0000 7f3199f94680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.892+0000 7f3199f94680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.892+0000 7f3199f94680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.892+0000 7f3199f94680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.892+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.896 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.893+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.894+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.894+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.897 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.894+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.904 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.900+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 zone default found 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 4 Realm: () 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 zonegroup default 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.901+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.902+0000 7f3199f94680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.902+0000 7f3199f94680 20 started sync module instance, tier type = 2023-10-05T19:58:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.902+0000 7f3199f94680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:51.914 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=0 2023-10-05T19:58:51.914 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=1 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=2 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=3 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=4 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=5 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=6 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 20 add_watcher() i=7 2023-10-05T19:58:51.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.912+0000 7f3199f94680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:51.916 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.914+0000 7f315e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:51.916 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.914+0000 7f3199f94680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:51.916 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.914+0000 7f3199f94680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:51.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.918+0000 7f3199f94680 5 note: GC not initialized 2023-10-05T19:58:51.922 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f30cd7fa700 20 reqs_thread_entry: start 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f30c6ffd700 20 reqs_thread_entry: start 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f30c57fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:22 2023 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f30c4ff9700 20 reqs_thread_entry: start 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 20 get_system_obj_state: rctx=0x7ffc4d2923b8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x564083d97240 s->prefetch_data=0 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T19:58:51.923 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 20 Read xattr: user.rgw.idtag 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.919+0000 7f3199f94680 20 rados->read ofs=0 len=0 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.920+0000 7f3199f94680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.920+0000 7f3199f94680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.920+0000 7f3199f94680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:51.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:51.920+0000 7f3199f94680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:53.445 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.442+0000 7f3199f94680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:58:53.445 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.442+0000 7f3199f94680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:53.445 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.442+0000 7f3199f94680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T19:58:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.443+0000 7f317e7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149698 cookie 94835091588480 notifier 4217 bl.length()=654 2023-10-05T19:58:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.443+0000 7f317e7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-10-05T19:58:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.443+0000 7f317e7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T19:58:53.447 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.445+0000 7f3199f94680 20 remove_watcher() i=0 2023-10-05T19:58:53.447 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.445+0000 7f3199f94680 2 removed watcher, disabling cache 2023-10-05T19:58:53.448 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.446+0000 7f3199f94680 20 remove_watcher() i=1 2023-10-05T19:58:53.451 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.449+0000 7f3199f94680 20 remove_watcher() i=2 2023-10-05T19:58:53.454 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.452+0000 7f3199f94680 20 remove_watcher() i=3 2023-10-05T19:58:53.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.454+0000 7f3199f94680 20 remove_watcher() i=4 2023-10-05T19:58:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.458+0000 7f3199f94680 20 remove_watcher() i=5 2023-10-05T19:58:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.459+0000 7f3199f94680 20 remove_watcher() i=6 2023-10-05T19:58:53.466 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.464+0000 7f3199f94680 20 remove_watcher() i=7 2023-10-05T19:58:53.467 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.465+0000 7f315e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.475 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-10-05T19:58:53.476 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key LFWOHPXQWFGHREZAHFEI --secret zg70MHGvZXMPrQro5XHuWzeq82WtyRbUHGN7XWGLxZwcSrNqB5ux3g== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-10-05T19:58:53.527 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:53.528 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:53.533 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.531+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 realm 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.536 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.536 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.532+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.536 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.533+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.536 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.533+0000 7f251127e680 20 searching for the correct realm 2023-10-05T19:58:53.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.545 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.545 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.541+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.545 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.542+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.545 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.542+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.542+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.548+0000 7f251127e680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.548+0000 7f251127e680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.548+0000 7f251127e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.548+0000 7f251127e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.548+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.551 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.552 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.549+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 zone default found 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 4 Realm: () 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 zonegroup default 2023-10-05T19:58:53.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 started sync module instance, tier type = 2023-10-05T19:58:53.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.550+0000 7f251127e680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:53.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=0 2023-10-05T19:58:53.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=1 2023-10-05T19:58:53.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=2 2023-10-05T19:58:53.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=3 2023-10-05T19:58:53.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=4 2023-10-05T19:58:53.570 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=5 2023-10-05T19:58:53.570 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=6 2023-10-05T19:58:53.570 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 20 add_watcher() i=7 2023-10-05T19:58:53.570 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.567+0000 7f251127e680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:53.571 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.569+0000 7f24d5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.571 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.569+0000 7f251127e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:53.572 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.569+0000 7f251127e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:53.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.573+0000 7f251127e680 5 note: GC not initialized 2023-10-05T19:58:53.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f244cff9700 20 reqs_thread_entry: start 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f24427fc700 20 reqs_thread_entry: start 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f2440ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:23 2023 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f2427fff700 20 reqs_thread_entry: start 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 20 get_system_obj_state: rctx=0x7ffc60703708 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55b5adabc240 s->prefetch_data=0 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 20 get_system_obj_state: rctx=0x7ffc607033f8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55b5adabc240 s->prefetch_data=0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.574+0000 7f251127e680 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 20 get_system_obj_state: rctx=0x7ffc607033f8 obj=default.rgw.meta:users.keys:LFWOHPXQWFGHREZAHFEI state=0x55b5adabc240 s->prefetch_data=0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 cache get: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI : miss 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI info.flags=0x0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 adding default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI to cache LRU end 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 20 get_system_obj_state: rctx=0x7ffc60702c88 obj=default.rgw.meta:users.keys:LFWOHPXQWFGHREZAHFEI state=0x55b5adabc240 s->prefetch_data=0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 cache get: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI : hit (negative entry) 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 20 get_system_obj_state: rctx=0x7ffc60703338 obj=default.rgw.meta:users.keys:LFWOHPXQWFGHREZAHFEI state=0x55b5adabc240 s->prefetch_data=0 2023-10-05T19:58:53.579 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.575+0000 7f251127e680 10 cache get: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI : hit (negative entry) 2023-10-05T19:58:53.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:58:53.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f251127e680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f251127e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f2502373700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 94238792785488 notifier 4223 bl.length()=640 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f2502373700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.576+0000 7f2502373700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f251127e680 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f251127e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f2501b72700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149698 cookie 94238792785488 notifier 4223 bl.length()=194 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f2501b72700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:53.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.578+0000 7f2501b72700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.582 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f251127e680 10 cache put: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI info.flags=0x7 2023-10-05T19:58:53.582 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f251127e680 10 moving default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI to cache LRU end 2023-10-05T19:58:53.582 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f251127e680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:LFWOHPXQWFGHREZAHFEI, ofs0, ns] 2023-10-05T19:58:53.582 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f2501b72700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149699 cookie 94238792769664 notifier 4223 bl.length()=186 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f2501b72700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI info.flags=0x7 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.579+0000 7f2501b72700 10 rgw watcher librados: moving default.rgw.meta+users.keys+LFWOHPXQWFGHREZAHFEI to cache LRU end 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "user_id": "bar.client.0", 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "display_name": "Mr. bar.client.0", 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "email": "bar.client.0+test@test.test", 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "suspended": 0, 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "max_buckets": 1000, 2023-10-05T19:58:53.583 INFO:teuthology.orchestra.run.smithi155.stdout: "subusers": [], 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "keys": [ 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "user": "bar.client.0", 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "access_key": "LFWOHPXQWFGHREZAHFEI", 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "secret_key": "zg70MHGvZXMPrQro5XHuWzeq82WtyRbUHGN7XWGLxZwcSrNqB5ux3g==" 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "swift_keys": [], 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": [ 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "user-policy", 2023-10-05T19:58:53.584 INFO:teuthology.orchestra.run.smithi155.stdout: "perm": "*" 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "op_mask": "read, write, delete", 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "default_placement": "", 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "default_storage_class": "", 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "placement_tags": [], 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "bucket_quota": { 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:53.585 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "user_quota": { 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "temp_url_keys": [], 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "rgw", 2023-10-05T19:58:53.586 INFO:teuthology.orchestra.run.smithi155.stdout: "mfa_ids": [] 2023-10-05T19:58:53.587 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-10-05T19:58:53.587 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:53.587 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.581+0000 7f251127e680 20 remove_watcher() i=0 2023-10-05T19:58:53.587 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.581+0000 7f251127e680 2 removed watcher, disabling cache 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.582+0000 7f251127e680 20 remove_watcher() i=1 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.582+0000 7f251127e680 20 remove_watcher() i=2 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.583+0000 7f251127e680 20 remove_watcher() i=3 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.584+0000 7f251127e680 20 remove_watcher() i=4 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.585+0000 7f251127e680 20 remove_watcher() i=5 2023-10-05T19:58:53.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.586+0000 7f251127e680 20 remove_watcher() i=6 2023-10-05T19:58:53.589 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.586+0000 7f251127e680 20 remove_watcher() i=7 2023-10-05T19:58:53.589 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.587+0000 7f24d5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.594 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 2728776227 --totp-seed USJK27UTNPOSOXFD4ESWJGRUOMPIH4MBTTVWDM34F2HNDONS2LARQQAFOS7QUZR7 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:58:53.642 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:53.643 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:53.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.645+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.649 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.646+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.649 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.646+0000 7f85e209d680 20 realm 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.646+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.647+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.648+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.648+0000 7f85e209d680 20 searching for the correct realm 2023-10-05T19:58:53.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.655+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.656+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.656+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.656+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.664 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.664 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.665 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.665 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.665 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.665 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.665 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.662+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 zone default found 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 4 Realm: () 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:53.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 zonegroup default 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.663+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.664+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.664+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.664+0000 7f85e209d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.664+0000 7f85e209d680 20 started sync module instance, tier type = 2023-10-05T19:58:53.668 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.664+0000 7f85e209d680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=0 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=1 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=2 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=3 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=4 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=5 2023-10-05T19:58:53.676 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.673+0000 7f85e209d680 20 add_watcher() i=6 2023-10-05T19:58:53.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.675+0000 7f85e209d680 20 add_watcher() i=7 2023-10-05T19:58:53.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.675+0000 7f85e209d680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:53.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.680+0000 7f85a67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.680+0000 7f85e209d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:53.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.680+0000 7f85e209d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:53.686 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.684+0000 7f85e209d680 5 note: GC not initialized 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.684+0000 7f85157fa700 20 reqs_thread_entry: start 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f850effd700 20 reqs_thread_entry: start 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f850d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:24 2023 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f850cff9700 20 reqs_thread_entry: start 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 20 get_system_obj_state: rctx=0x7ffe7c47c3c8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x5624f7184240 s->prefetch_data=0 2023-10-05T19:58:53.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 20 Read xattr: user.rgw.idtag 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.685+0000 7f85e209d680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.689 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.686+0000 7f85e209d680 20 rados_obj.operate() r=0 bl.length=455 2023-10-05T19:58:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.686+0000 7f85e209d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T19:58:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.686+0000 7f85e209d680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.686+0000 7f85e209d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85e209d680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85e209d680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85e209d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85c6ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149699 cookie 94716765996176 notifier 4229 bl.length()=654 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85c6ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-10-05T19:58:53.691 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.688+0000 7f85c6ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T19:58:53.692 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.690+0000 7f85e209d680 20 remove_watcher() i=0 2023-10-05T19:58:53.692 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.690+0000 7f85e209d680 2 removed watcher, disabling cache 2023-10-05T19:58:53.693 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.691+0000 7f85e209d680 20 remove_watcher() i=1 2023-10-05T19:58:53.694 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.691+0000 7f85e209d680 20 remove_watcher() i=2 2023-10-05T19:58:53.695 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.692+0000 7f85e209d680 20 remove_watcher() i=3 2023-10-05T19:58:53.695 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.693+0000 7f85e209d680 20 remove_watcher() i=4 2023-10-05T19:58:53.696 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.694+0000 7f85e209d680 20 remove_watcher() i=5 2023-10-05T19:58:53.697 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.695+0000 7f85e209d680 20 remove_watcher() i=6 2023-10-05T19:58:53.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.695+0000 7f85e209d680 20 remove_watcher() i=7 2023-10-05T19:58:53.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.696+0000 7f85a67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.704 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-10-05T19:58:53.704 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key XCWEZXIPPXCDUXORUFKW --secret vVE8x5wDtiirquJhMWMCd1WKP4GUUCPiTN8mD586I7dnzMMHsgo/Ew== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-10-05T19:58:53.752 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:53.752 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:53.757 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.755+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.758 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.758 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 realm 2023-10-05T19:58:53.759 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.759 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.759 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:53.759 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.756+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.757+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.757+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.757+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.760 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.757+0000 7fd2e6388680 20 searching for the correct realm 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.767 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.764+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.768 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.768 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.768 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.768 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.765+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.766+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.776 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.772+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 zone default found 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 4 Realm: () 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:53.777 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 zonegroup default 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 started sync module instance, tier type = 2023-10-05T19:58:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.773+0000 7fd2e6388680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:53.800 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=0 2023-10-05T19:58:53.800 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=1 2023-10-05T19:58:53.800 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=2 2023-10-05T19:58:53.800 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=3 2023-10-05T19:58:53.801 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=4 2023-10-05T19:58:53.801 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=5 2023-10-05T19:58:53.801 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.798+0000 7fd2e6388680 20 add_watcher() i=6 2023-10-05T19:58:53.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.800+0000 7fd2e6388680 20 add_watcher() i=7 2023-10-05T19:58:53.802 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.800+0000 7fd2e6388680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:53.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.803+0000 7fd2aa7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.803+0000 7fd2e6388680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:53.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.803+0000 7fd2e6388680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.807+0000 7fd2e6388680 5 note: GC not initialized 2023-10-05T19:58:53.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2197fa700 20 reqs_thread_entry: start 2023-10-05T19:58:53.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd212ffd700 20 reqs_thread_entry: start 2023-10-05T19:58:53.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2117fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:23 2023 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd210ff9700 20 reqs_thread_entry: start 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 20 get_system_obj_state: rctx=0x7ffd607b93f8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5586b7d40240 s->prefetch_data=0 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 20 get_system_obj_state: rctx=0x7ffd607b90e8 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x5586b7d40240 s->prefetch_data=0 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.808+0000 7fd2e6388680 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-10-05T19:58:53.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 20 get_system_obj_state: rctx=0x7ffd607b90e8 obj=default.rgw.meta:users.keys:XCWEZXIPPXCDUXORUFKW state=0x5586b7d40240 s->prefetch_data=0 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 cache get: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW : miss 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW info.flags=0x0 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 adding default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW to cache LRU end 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 20 get_system_obj_state: rctx=0x7ffd607b8978 obj=default.rgw.meta:users.keys:XCWEZXIPPXCDUXORUFKW state=0x5586b7d40240 s->prefetch_data=0 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 cache get: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW : hit (negative entry) 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 20 get_system_obj_state: rctx=0x7ffd607b9028 obj=default.rgw.meta:users.keys:XCWEZXIPPXCDUXORUFKW state=0x5586b7d40240 s->prefetch_data=0 2023-10-05T19:58:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.809+0000 7fd2e6388680 10 cache get: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW : hit (negative entry) 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2e6388680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2e6388680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2caffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149700 cookie 94037099744064 notifier 4235 bl.length()=714 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2caffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:58:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.810+0000 7fd2caffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2e6388680 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2e6388680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T19:58:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2ca7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149701 cookie 94037099746368 notifier 4235 bl.length()=224 2023-10-05T19:58:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2ca7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-10-05T19:58:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.815+0000 7fd2ca7fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-10-05T19:58:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.816+0000 7fd2e6388680 10 cache put: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW info.flags=0x7 2023-10-05T19:58:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.816+0000 7fd2e6388680 10 moving default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW to cache LRU end 2023-10-05T19:58:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.816+0000 7fd2e6388680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:XCWEZXIPPXCDUXORUFKW, ofs0, ns] 2023-10-05T19:58:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.817+0000 7fd2caffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149702 cookie 94037099744064 notifier 4235 bl.length()=201 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.817+0000 7fd2caffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW info.flags=0x7 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.817+0000 7fd2caffd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+XCWEZXIPPXCDUXORUFKW to cache LRU end 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "user_id": "testx$tenanteduser.client.0", 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "suspended": 0, 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "max_buckets": 1000, 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "subusers": [], 2023-10-05T19:58:53.820 INFO:teuthology.orchestra.run.smithi155.stdout: "keys": [ 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "user": "testx$tenanteduser.client.0", 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "access_key": "XCWEZXIPPXCDUXORUFKW", 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "secret_key": "vVE8x5wDtiirquJhMWMCd1WKP4GUUCPiTN8mD586I7dnzMMHsgo/Ew==" 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "swift_keys": [], 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": [ 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "user-policy", 2023-10-05T19:58:53.821 INFO:teuthology.orchestra.run.smithi155.stdout: "perm": "*" 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "op_mask": "read, write, delete", 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "default_placement": "", 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "default_storage_class": "", 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "placement_tags": [], 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "bucket_quota": { 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:53.822 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "user_quota": { 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "enabled": false, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "check_on_raw": false, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size": -1, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "max_size_kb": 0, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "max_objects": -1 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "temp_url_keys": [], 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "type": "rgw", 2023-10-05T19:58:53.823 INFO:teuthology.orchestra.run.smithi155.stdout: "mfa_ids": [] 2023-10-05T19:58:53.824 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-10-05T19:58:53.824 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:58:53.826 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.824+0000 7fd2e6388680 20 remove_watcher() i=0 2023-10-05T19:58:53.826 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.824+0000 7fd2e6388680 2 removed watcher, disabling cache 2023-10-05T19:58:53.827 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.824+0000 7fd2e6388680 20 remove_watcher() i=1 2023-10-05T19:58:53.829 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.827+0000 7fd2e6388680 20 remove_watcher() i=2 2023-10-05T19:58:53.832 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.830+0000 7fd2e6388680 20 remove_watcher() i=3 2023-10-05T19:58:53.834 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.832+0000 7fd2e6388680 20 remove_watcher() i=4 2023-10-05T19:58:53.835 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.833+0000 7fd2e6388680 20 remove_watcher() i=5 2023-10-05T19:58:53.837 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.835+0000 7fd2e6388680 20 remove_watcher() i=6 2023-10-05T19:58:53.838 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.836+0000 7fd2e6388680 20 remove_watcher() i=7 2023-10-05T19:58:53.839 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.836+0000 7fd2aa7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.844 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 3701687145 --totp-seed CZHZO62AWWBFOHAHWZPMCDQKCV42D3ICPZHEUGLRY5JI5RW5AMXM3BPRRHH5ISAE --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-10-05T19:58:53.893 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:58:53.893 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:58:53.898 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.896+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.899 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 realm 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.900 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.897+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.901 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.898+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.901 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.898+0000 7fe0cc228680 20 searching for the correct realm 2023-10-05T19:58:53.908 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.908 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.908 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.908 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.905+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.906+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.906+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.906+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.909 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.906+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:58:53.915 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.912+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 zone default found 2023-10-05T19:58:53.917 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 4 Realm: () 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 zonegroup default 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.913+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.914+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.914+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.914+0000 7fe0cc228680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:58:53.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.914+0000 7fe0cc228680 20 started sync module instance, tier type = 2023-10-05T19:58:53.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.914+0000 7fe0cc228680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T19:58:53.931 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=0 2023-10-05T19:58:53.931 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=1 2023-10-05T19:58:53.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=2 2023-10-05T19:58:53.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=3 2023-10-05T19:58:53.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=4 2023-10-05T19:58:53.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=5 2023-10-05T19:58:53.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.929+0000 7fe0cc228680 20 add_watcher() i=6 2023-10-05T19:58:53.933 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.931+0000 7fe0cc228680 20 add_watcher() i=7 2023-10-05T19:58:53.934 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.931+0000 7fe0cc228680 2 all 8 watchers are set, enabling cache 2023-10-05T19:58:53.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.932+0000 7fe00bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.932+0000 7fe0cc228680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:58:53.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.932+0000 7fe0cc228680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:58:53.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.936+0000 7fe0cc228680 5 note: GC not initialized 2023-10-05T19:58:53.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fdffaffd700 20 reqs_thread_entry: start 2023-10-05T19:58:53.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fdff8ff9700 20 reqs_thread_entry: start 2023-10-05T19:58:53.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fe0cc228680 20 init_complete bucket index max shards: 11 2023-10-05T19:58:53.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fe0cc228680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:58:53.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fdff2ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:59:24 2023 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fdff27fc700 20 reqs_thread_entry: start 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fe0cc228680 20 get_system_obj_state: rctx=0x7ffe2aa05b28 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x56336de79240 s->prefetch_data=0 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.937+0000 7fe0cc228680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:58:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 20 Read xattr: user.rgw.idtag 2023-10-05T19:58:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:58:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 20 rados->read ofs=0 len=0 2023-10-05T19:58:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 20 rados_obj.operate() r=0 bl.length=514 2023-10-05T19:58:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T19:58:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.938+0000 7fe0cc228680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:58:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.940+0000 7fe0cc228680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.940+0000 7fe0cc228680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.940+0000 7fe0cc228680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.941+0000 7fe0bd31d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149703 cookie 94778888849152 notifier 4241 bl.length()=728 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.941+0000 7fe0bd31d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.941+0000 7fe0bd31d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T19:58:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.942+0000 7fe0cc228680 20 remove_watcher() i=0 2023-10-05T19:58:53.948 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.942+0000 7fe0cc228680 2 removed watcher, disabling cache 2023-10-05T19:58:53.948 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.944+0000 7fe0cc228680 20 remove_watcher() i=1 2023-10-05T19:58:53.948 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.945+0000 7fe0cc228680 20 remove_watcher() i=2 2023-10-05T19:58:53.949 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.947+0000 7fe0cc228680 20 remove_watcher() i=3 2023-10-05T19:58:53.950 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.947+0000 7fe0cc228680 20 remove_watcher() i=4 2023-10-05T19:58:53.951 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.949+0000 7fe0cc228680 20 remove_watcher() i=5 2023-10-05T19:58:53.952 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.950+0000 7fe0cc228680 20 remove_watcher() i=6 2023-10-05T19:58:53.953 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.951+0000 7fe0cc228680 20 remove_watcher() i=7 2023-10-05T19:58:53.954 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T19:58:53.952+0000 7fe00bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:58:53.959 INFO:tasks.s3tests:Configuring s3-tests... 2023-10-05T19:58:53.959 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest/s3-tests-client.0 && ./bootstrap 2023-10-05T19:59:00.783 INFO:teuthology.orchestra.run.smithi155.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2023-10-05T19:59:01.143 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-10-05T19:59:02.050 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:06 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:59:03.872 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-10-05T19:59:03.873 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: libevent-devel x86_64 2.1.8-5.el8 rhel-8-for-x86_64-appstream-rpms 104 k 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: libffi-devel x86_64 3.1-22.el8 rhel-8-for-x86_64-baseos-rpms 29 k 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: libxml2-devel x86_64 2.9.7-9.el8_4.2 rhel-8-for-x86_64-appstream-rpms 1.0 M 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt-devel x86_64 1.1.32-6.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-10-05T19:59:03.874 INFO:teuthology.orchestra.run.smithi155.stdout: cmake-filesystem x86_64 3.18.2-11.el8_4 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: libgcrypt-devel x86_64 1.8.5-4.el8 rhel-8-for-x86_64-baseos-rpms 149 k 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: libgpg-error-devel 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: x86_64 1.31-1.el8 rhel-8-for-x86_64-baseos-rpms 64 k 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: xz-devel x86_64 5.2.4-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout:Install 9 Packages 2023-10-05T19:59:03.875 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:03.876 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.9 M 2023-10-05T19:59:03.876 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 12 M 2023-10-05T19:59:03.876 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-10-05T19:59:03.945 INFO:teuthology.orchestra.run.smithi155.stdout:(1/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 4.6 MB/s | 323 kB 00:00 2023-10-05T19:59:03.963 INFO:teuthology.orchestra.run.smithi155.stdout:(2/9): libxml2-devel-2.9.7-9.el8_4.2.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2023-10-05T19:59:03.964 INFO:teuthology.orchestra.run.smithi155.stdout:(3/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 1.2 MB/s | 104 kB 00:00 2023-10-05T19:59:04.007 INFO:teuthology.orchestra.run.smithi155.stdout:(4/9): cmake-filesystem-3.18.2-11.el8_4.x86_64. 722 kB/s | 44 kB 00:00 2023-10-05T19:59:04.018 INFO:teuthology.orchestra.run.smithi155.stdout:(5/9): libffi-devel-3.1-22.el8.x86_64.rpm 526 kB/s | 29 kB 00:00 2023-10-05T19:59:04.035 INFO:teuthology.orchestra.run.smithi155.stdout:(6/9): xz-devel-5.2.4-3.el8.x86_64.rpm 885 kB/s | 62 kB 00:00 2023-10-05T19:59:04.069 INFO:teuthology.orchestra.run.smithi155.stdout:(7/9): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-10-05T19:59:04.076 INFO:teuthology.orchestra.run.smithi155.stdout:(8/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 923 kB/s | 64 kB 00:00 2023-10-05T19:59:04.077 INFO:teuthology.orchestra.run.smithi155.stdout:(9/9): libgcrypt-devel-1.8.5-4.el8.x86_64.rpm 2.5 MB/s | 149 kB 00:00 2023-10-05T19:59:04.078 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-10-05T19:59:04.078 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.1 MB/s | 1.9 MB 00:00 2023-10-05T19:59:04.086 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-10-05T19:59:04.099 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-10-05T19:59:04.099 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-10-05T19:59:04.247 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-10-05T19:59:04.252 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-10-05T19:59:04.651 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-10-05T19:59:04.834 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-10-05T19:59:04.876 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/9 2023-10-05T19:59:05.082 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-10-05T19:59:05.097 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgcrypt-devel-1.8.5-4.el8.x86_64 2/9 2023-10-05T19:59:05.230 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 3/9 2023-10-05T19:59:05.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xz-devel-5.2.4-3.el8.x86_64 4/9 2023-10-05T19:59:05.788 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cmake-filesystem-3.18.2-11.el8_4.x86_64 5/9 2023-10-05T19:59:06.086 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libxml2-devel-2.9.7-9.el8_4.2.x86_64 6/9 2023-10-05T19:59:06.297 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 7/9 2023-10-05T19:59:06.458 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libffi-devel-3.1-22.el8.x86_64 8/9 2023-10-05T19:59:06.490 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libffi-devel-3.1-22.el8.x86_64 8/9 2023-10-05T19:59:06.699 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 9/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libevent-devel-2.1.8-5.el8.x86_64 9/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 1/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 2/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libxml2-devel-2.9.7-9.el8_4.2.x86_64 3/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cmake-filesystem-3.18.2-11.el8_4.x86_64 4/9 2023-10-05T19:59:06.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libffi-devel-3.1-22.el8.x86_64 5/9 2023-10-05T19:59:06.999 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xz-devel-5.2.4-3.el8.x86_64 6/9 2023-10-05T19:59:06.999 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/9 2023-10-05T19:59:06.999 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgcrypt-devel-1.8.5-4.el8.x86_64 8/9 2023-10-05T19:59:08.133 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 9/9 2023-10-05T19:59:08.133 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:12 ago on Thu 05 Oct 2023 07:53:56 PM UTC. 2023-10-05T19:59:10.147 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: cmake-filesystem-3.18.2-11.el8_4.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: libffi-devel-3.1-22.el8.x86_64 libgcrypt-devel-1.8.5-4.el8.x86_64 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-9.el8_4.2.x86_64 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-3.el8.x86_64 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-05T19:59:10.249 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:10.250 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-10-05T19:59:10.712 INFO:teuthology.orchestra.run.smithi155.stdout:Already using interpreter /usr/bin/python3.6 2023-10-05T19:59:10.712 INFO:teuthology.orchestra.run.smithi155.stdout:Using base prefix '/usr' 2023-10-05T19:59:10.712 INFO:teuthology.orchestra.run.smithi155.stdout:New python executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python3.6 2023-10-05T19:59:10.712 INFO:teuthology.orchestra.run.smithi155.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python 2023-10-05T19:59:13.862 INFO:teuthology.orchestra.run.smithi155.stdout:Installing setuptools, pip, wheel...done. 2023-10-05T19:59:14.201 INFO:teuthology.orchestra.run.smithi155.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-10-05T19:59:15.457 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting setuptools==44.1.0 2023-10-05T19:59:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-10-05T19:59:15.677 INFO:teuthology.orchestra.run.smithi155.stdout:Installing collected packages: setuptools 2023-10-05T19:59:15.677 INFO:teuthology.orchestra.run.smithi155.stdout: Attempting uninstall: setuptools 2023-10-05T19:59:15.677 INFO:teuthology.orchestra.run.smithi155.stdout: Found existing installation: setuptools 59.6.0 2023-10-05T19:59:15.723 INFO:teuthology.orchestra.run.smithi155.stdout: Uninstalling setuptools-59.6.0: 2023-10-05T19:59:15.731 INFO:teuthology.orchestra.run.smithi155.stdout: Successfully uninstalled setuptools-59.6.0 2023-10-05T19:59:15.985 INFO:teuthology.orchestra.run.smithi155.stdout:Successfully installed setuptools-44.1.0 2023-10-05T19:59:16.583 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting PyYAML 2023-10-05T19:59:16.637 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2023-10-05T19:59:16.802 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting nose>=1.0.0 2023-10-05T19:59:16.820 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-10-05T19:59:16.964 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting boto>=2.6.0 2023-10-05T19:59:16.985 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-10-05T19:59:17.958 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting boto3>=1.0.0 2023-10-05T19:59:17.976 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-10-05T19:59:19.058 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting botocore<1.28.0 2023-10-05T19:59:19.086 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-10-05T19:59:19.356 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting munch>=2.0.0 2023-10-05T19:59:19.385 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-10-05T19:59:19.886 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting gevent>=1.0 2023-10-05T19:59:19.934 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-10-05T19:59:20.185 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting isodate>=0.4.4 2023-10-05T19:59:20.206 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-10-05T19:59:20.338 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting requests>=2.23.0 2023-10-05T19:59:20.355 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2023-10-05T19:59:20.557 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting pytz>=2011k 2023-10-05T19:59:20.575 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-10-05T19:59:20.697 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting httplib2 2023-10-05T19:59:20.715 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-10-05T19:59:21.254 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting lxml 2023-10-05T19:59:21.306 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl (7.4 MB) 2023-10-05T19:59:21.534 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-10-05T19:59:21.550 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-10-05T19:59:21.665 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-10-05T19:59:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-10-05T19:59:21.916 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-10-05T19:59:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-10-05T19:59:22.057 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting urllib3<1.27,>=1.25.4 2023-10-05T19:59:22.083 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading urllib3-1.26.17-py2.py3-none-any.whl (143 kB) 2023-10-05T19:59:22.300 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting importlib-metadata>=1.7.0 2023-10-05T19:59:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2023-10-05T19:59:22.449 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting zope.event 2023-10-05T19:59:22.468 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-10-05T19:59:22.975 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting greenlet>=2.0.0 2023-10-05T19:59:22.994 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2023-10-05T19:59:23.409 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting zope.interface 2023-10-05T19:59:23.429 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-10-05T19:59:23.475 INFO:teuthology.orchestra.run.smithi155.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-10-05T19:59:23.542 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting six 2023-10-05T19:59:23.559 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-10-05T19:59:23.656 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting certifi>=2017.4.17 2023-10-05T19:59:23.675 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading certifi-2023.7.22-py3-none-any.whl (158 kB) 2023-10-05T19:59:23.794 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting idna<4,>=2.5 2023-10-05T19:59:23.810 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-10-05T19:59:24.101 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting charset-normalizer~=2.0.0 2023-10-05T19:59:24.119 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2023-10-05T19:59:24.262 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-10-05T19:59:24.280 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-10-05T19:59:24.436 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting typing-extensions>=3.6.4 2023-10-05T19:59:24.467 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2023-10-05T19:59:24.573 INFO:teuthology.orchestra.run.smithi155.stdout:Collecting zipp>=0.5 2023-10-05T19:59:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2023-10-05T19:59:25.089 INFO:teuthology.orchestra.run.smithi155.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2023-10-05T19:59:28.729 INFO:teuthology.orchestra.run.smithi155.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.7.22 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.4 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.17 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2023-10-05T19:59:29.043 INFO:teuthology.orchestra.run.smithi155.stdout:running develop 2023-10-05T19:59:29.065 INFO:teuthology.orchestra.run.smithi155.stdout:running egg_info 2023-10-05T19:59:29.065 INFO:teuthology.orchestra.run.smithi155.stdout:creating s3tests.egg-info 2023-10-05T19:59:29.066 INFO:teuthology.orchestra.run.smithi155.stdout:writing s3tests.egg-info/PKG-INFO 2023-10-05T19:59:29.066 INFO:teuthology.orchestra.run.smithi155.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-10-05T19:59:29.066 INFO:teuthology.orchestra.run.smithi155.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-10-05T19:59:29.066 INFO:teuthology.orchestra.run.smithi155.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-10-05T19:59:29.066 INFO:teuthology.orchestra.run.smithi155.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:59:29.075 INFO:teuthology.orchestra.run.smithi155.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:59:29.076 INFO:teuthology.orchestra.run.smithi155.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-10-05T19:59:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:running build_ext 2023-10-05T19:59:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:Creating /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2023-10-05T19:59:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-10-05T19:59:29.092 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:Installed /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T19:59:29.093 INFO:teuthology.orchestra.run.smithi155.stdout:Processing dependencies for s3tests==0.0.1 2023-10-05T19:59:29.204 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for isodate==0.6.1 2023-10-05T19:59:29.204 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: isodate 0.6.1 2023-10-05T19:59:29.204 INFO:teuthology.orchestra.run.smithi155.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for gevent==22.10.2 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: gevent 22.10.2 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.205 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for munch==4.0.0 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: munch 4.0.0 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for PyYAML==6.0.1 2023-10-05T19:59:29.210 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: PyYAML 6.0.1 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for boto3==1.23.10 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: boto3 1.23.10 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Adding boto3 1.23.10 to easy-install.pth file 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for boto==2.49.0 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: boto 2.49.0 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-10-05T19:59:29.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for six==1.16.0 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: six 1.16.0 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Adding six 1.16.0 to easy-install.pth file 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for setuptools==44.1.0 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: setuptools 44.1.0 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-10-05T19:59:29.212 INFO:teuthology.orchestra.run.smithi155.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:59:29.213 INFO:teuthology.orchestra.run.smithi155.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin 2023-10-05T19:59:29.213 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.213 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.216 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for zope.event==4.6 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: zope.event 4.6 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for greenlet==2.0.2 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: greenlet 2.0.2 2023-10-05T19:59:29.217 INFO:teuthology.orchestra.run.smithi155.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for zope.interface==5.5.2 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: zope.interface 5.5.2 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for importlib-metadata==4.8.3 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: importlib-metadata 4.8.3 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2023-10-05T19:59:29.218 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for s3transfer==0.5.2 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: s3transfer 0.5.2 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for botocore==1.26.10 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: botocore 1.26.10 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout:Adding botocore 1.26.10 to easy-install.pth file 2023-10-05T19:59:29.219 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for jmespath==0.10.0 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: jmespath 0.10.0 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for typing-extensions==4.1.1 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: typing-extensions 4.1.1 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.220 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.222 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for zipp==3.6.0 2023-10-05T19:59:29.222 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: zipp 3.6.0 2023-10-05T19:59:29.222 INFO:teuthology.orchestra.run.smithi155.stdout:Adding zipp 3.6.0 to easy-install.pth file 2023-10-05T19:59:29.222 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.222 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for python-dateutil==2.8.2 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: python-dateutil 2.8.2 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Searching for urllib3==1.26.17 2023-10-05T19:59:29.223 INFO:teuthology.orchestra.run.smithi155.stdout:Best match: urllib3 1.26.17 2023-10-05T19:59:29.224 INFO:teuthology.orchestra.run.smithi155.stdout:Adding urllib3 1.26.17 to easy-install.pth file 2023-10-05T19:59:29.224 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T19:59:29.224 INFO:teuthology.orchestra.run.smithi155.stdout:Using /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages 2023-10-05T19:59:29.224 INFO:teuthology.orchestra.run.smithi155.stdout:Finished processing dependencies for s3tests==0.0.1 2023-10-05T19:59:29.238 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:59:29.238 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-10-05T19:59:29.253 INFO:tasks.s3tests:Configuring boto... 2023-10-05T19:59:29.254 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-10-05T19:59:29.254 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T19:59:29.308 DEBUG:teuthology.orchestra.run.smithi155:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-10-05T19:59:29.822 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-10-05T19:59:29.880 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-10-05T19:59:29.943 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-10-05T19:59:30.002 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-10-05T19:59:30.057 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-10-05T19:59:30.079 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-10-05T19:59:30.153 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-10-05T19:59:30.166 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-10-05T19:59:30.207 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-10-05T19:59:30.250 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-10-05T19:59:30.301 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-10-05T19:59:30.356 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-10-05T19:59:30.411 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:59:30.468 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-10-05T19:59:30.470 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-10-05T19:59:30.515 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-10-05T19:59:30.516 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-10-05T19:59:30.516 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-10-05T19:59:30.516 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-10-05T19:59:30.516 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:59:30.516 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:59:30.517 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:59:30.517 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:59:30.517 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:59:30.517 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:59:30.517 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:59:30.518 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:59:30.518 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:59:30.518 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:59:30.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-10-05T19:59:30.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-10-05T19:59:30.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-10-05T19:59:30.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-10-05T19:59:30.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-10-05T19:59:30.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-10-05T19:59:30.521 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-10-05T19:59:30.521 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-10-05T19:59:30.521 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-10-05T19:59:30.521 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-10-05T19:59:30.521 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-10-05T19:59:30.522 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-10-05T19:59:30.523 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-10-05T19:59:30.523 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-10-05T19:59:30.663 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-10-05T19:59:30.664 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-10-05T19:59:30.664 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-10-05T19:59:30.664 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-10-05T19:59:30.664 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-10-05T19:59:30.665 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-10-05T19:59:30.665 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-10-05T19:59:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-10-05T19:59:30.818 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-10-05T19:59:32.580 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-10-05T19:59:32.675 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-10-05T19:59:32.745 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-10-05T19:59:32.776 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-10-05T19:59:32.873 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:59:32.874 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-10-05T19:59:32.875 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-10-05T19:59:32.876 INFO:teuthology.orchestra.run.smithi155.stderr:SKIP 2023-10-05T19:59:34.573 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-10-05T19:59:34.573 INFO:teuthology.orchestra.run.smithi155.stderr:/home/ubuntu/cephtest/s3-tests-client.0/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-10-05T19:59:34.573 INFO:teuthology.orchestra.run.smithi155.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-10-05T19:59:34.598 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-10-05T19:59:46.265 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-10-05T19:59:46.299 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-10-05T19:59:46.347 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-10-05T19:59:46.394 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-10-05T19:59:46.431 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-10-05T19:59:46.494 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-10-05T19:59:46.524 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-10-05T19:59:46.568 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-10-05T19:59:46.602 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-10-05T19:59:46.635 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-10-05T19:59:46.659 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-10-05T19:59:46.675 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-10-05T19:59:46.690 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-10-05T19:59:46.698 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-10-05T19:59:46.723 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-10-05T19:59:46.755 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:59:46.794 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-10-05T19:59:46.821 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:59:46.847 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-10-05T19:59:46.873 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:59:46.897 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:59:46.922 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-10-05T19:59:46.937 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-10-05T19:59:46.951 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-10-05T19:59:46.960 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-10-05T19:59:46.969 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-10-05T19:59:46.977 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-10-05T19:59:46.985 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-10-05T19:59:46.993 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-10-05T19:59:47.079 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-10-05T19:59:47.173 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-10-05T19:59:47.267 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-10-05T19:59:47.363 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-10-05T19:59:47.453 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-10-05T19:59:47.537 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-10-05T19:59:47.596 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-10-05T19:59:47.648 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-10-05T19:59:47.704 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-10-05T19:59:47.835 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-10-05T19:59:47.956 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-10-05T19:59:47.991 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:59:48.025 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-10-05T19:59:48.080 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-10-05T19:59:48.141 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-10-05T19:59:48.296 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-10-05T19:59:48.415 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-10-05T19:59:48.471 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-10-05T19:59:48.530 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-10-05T19:59:48.590 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-10-05T19:59:48.645 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-10-05T19:59:48.698 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-10-05T19:59:48.759 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-10-05T19:59:48.819 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-10-05T19:59:48.873 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-10-05T19:59:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-10-05T19:59:48.987 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-10-05T19:59:49.041 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-10-05T19:59:49.121 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-10-05T19:59:49.169 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-10-05T19:59:49.221 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-10-05T19:59:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-10-05T19:59:49.327 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-10-05T19:59:49.381 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-10-05T19:59:56.250 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-10-05T19:59:56.299 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-10-05T19:59:56.374 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-10-05T19:59:56.421 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-10-05T19:59:56.469 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-10-05T19:59:56.520 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-10-05T19:59:56.574 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-10-05T19:59:56.627 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-10-05T19:59:56.681 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-10-05T19:59:56.728 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-10-05T19:59:56.778 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-10-05T19:59:56.832 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-10-05T19:59:56.886 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-10-05T19:59:56.943 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-10-05T19:59:56.999 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-10-05T19:59:57.056 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-10-05T19:59:57.117 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-10-05T19:59:57.175 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:59:57.234 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-10-05T19:59:57.318 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:59:57.381 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-10-05T19:59:57.434 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:59:57.489 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-10-05T19:59:57.589 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-10-05T19:59:57.694 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-10-05T19:59:57.755 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-10-05T19:59:57.810 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-10-05T19:59:57.865 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-10-05T19:59:57.927 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-10-05T19:59:58.154 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-10-05T19:59:58.192 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-10-05T19:59:58.588 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-10-05T19:59:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-10-05T19:59:59.065 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-10-05T19:59:59.124 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-10-05T19:59:59.181 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-10-05T19:59:59.244 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-10-05T19:59:59.344 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-10-05T19:59:59.446 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-10-05T19:59:59.506 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-10-05T19:59:59.568 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-10-05T19:59:59.624 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-10-05T19:59:59.679 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-10-05T19:59:59.738 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-10-05T19:59:59.795 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-10-05T20:00:00.025 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-10-05T20:00:00.366 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-10-05T20:00:00.400 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-10-05T20:00:00.438 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-10-05T20:00:00.462 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-10-05T20:00:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-10-05T20:00:00.497 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-10-05T20:00:00.506 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-10-05T20:00:00.515 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-10-05T20:00:00.558 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-10-05T20:00:00.712 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-10-05T20:00:00.721 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-10-05T20:00:00.816 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-10-05T20:00:00.846 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-10-05T20:00:00.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-10-05T20:00:01.104 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-10-05T20:00:01.330 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-10-05T20:00:08.491 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-10-05T20:00:15.945 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-10-05T20:00:15.979 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-10-05T20:00:16.011 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-10-05T20:00:16.046 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-10-05T20:00:16.078 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-10-05T20:00:16.209 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-10-05T20:00:16.241 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-10-05T20:00:16.280 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-10-05T20:00:16.332 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-10-05T20:00:16.375 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-10-05T20:00:16.450 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-10-05T20:00:16.519 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-10-05T20:00:16.552 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-10-05T20:00:16.618 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-10-05T20:00:16.639 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-10-05T20:00:16.666 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-10-05T20:00:16.693 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-10-05T20:00:16.758 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-10-05T20:00:16.794 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-10-05T20:00:16.826 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-10-05T20:00:16.859 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-10-05T20:00:16.893 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-10-05T20:00:16.964 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-10-05T20:00:17.013 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-10-05T20:00:17.035 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-10-05T20:00:17.063 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-10-05T20:00:17.088 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-10-05T20:00:17.112 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-10-05T20:00:17.140 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-10-05T20:00:17.175 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-10-05T20:00:17.199 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-10-05T20:00:17.221 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-10-05T20:00:17.262 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-10-05T20:00:17.286 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-10-05T20:00:17.313 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-10-05T20:00:17.344 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-10-05T20:00:17.372 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-10-05T20:00:17.398 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-10-05T20:00:17.427 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-10-05T20:00:17.451 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-10-05T20:00:17.475 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-10-05T20:00:17.500 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-10-05T20:00:17.572 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-10-05T20:00:17.651 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-10-05T20:00:17.686 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-10-05T20:00:17.760 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-10-05T20:00:17.841 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-10-05T20:00:18.883 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-10-05T20:00:18.965 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-10-05T20:00:19.040 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-10-05T20:00:19.165 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-10-05T20:00:19.329 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-10-05T20:00:19.456 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-10-05T20:00:19.573 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-10-05T20:00:19.697 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-10-05T20:00:19.858 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-10-05T20:00:19.955 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-10-05T20:00:20.115 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-10-05T20:00:20.148 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-10-05T20:00:20.218 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-10-05T20:00:20.290 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-10-05T20:00:20.334 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-10-05T20:00:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-10-05T20:00:20.367 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-10-05T20:00:20.429 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-10-05T20:00:20.463 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-10-05T20:00:20.499 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-10-05T20:00:20.538 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-10-05T20:00:20.571 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-10-05T20:00:20.609 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-10-05T20:00:20.642 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-10-05T20:00:20.749 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-10-05T20:00:20.830 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-10-05T20:00:20.864 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-10-05T20:00:20.898 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-10-05T20:00:20.935 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-10-05T20:00:20.971 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-10-05T20:00:21.014 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-10-05T20:00:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-10-05T20:00:21.109 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-10-05T20:00:21.141 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-10-05T20:00:21.152 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-10-05T20:00:21.163 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-10-05T20:00:21.173 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-10-05T20:00:21.183 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-10-05T20:00:21.210 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-10-05T20:00:21.275 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-10-05T20:00:21.308 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-10-05T20:00:21.377 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-10-05T20:00:21.440 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-10-05T20:00:21.467 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-10-05T20:00:21.527 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-10-05T20:00:21.636 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-10-05T20:00:21.698 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-10-05T20:00:21.763 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-10-05T20:00:21.843 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-10-05T20:00:21.921 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-10-05T20:00:22.047 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-10-05T20:00:22.124 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-10-05T20:00:22.197 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-10-05T20:00:22.320 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-10-05T20:00:22.445 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-10-05T20:00:22.541 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-10-05T20:00:22.592 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-10-05T20:00:22.619 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-10-05T20:00:22.743 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-10-05T20:00:22.866 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-10-05T20:00:22.992 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-10-05T20:00:23.118 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-10-05T20:00:23.246 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-10-05T20:00:23.366 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-10-05T20:00:23.503 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-10-05T20:00:23.616 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-10-05T20:00:23.730 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-10-05T20:00:23.850 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-10-05T20:00:23.889 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-10-05T20:00:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-10-05T20:00:24.068 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-10-05T20:00:24.153 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-10-05T20:00:24.230 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-10-05T20:00:24.264 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-10-05T20:00:24.389 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-10-05T20:00:24.598 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-10-05T20:00:24.832 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-10-05T20:00:25.039 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-10-05T20:00:25.255 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-10-05T20:00:25.468 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-10-05T20:00:25.678 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-10-05T20:00:25.885 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-10-05T20:00:26.097 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-10-05T20:00:26.309 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-10-05T20:00:26.477 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-10-05T20:00:26.648 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-10-05T20:00:26.847 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-10-05T20:00:27.034 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-10-05T20:00:27.189 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-10-05T20:00:27.195 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-10-05T20:00:27.204 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-10-05T20:00:27.212 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-10-05T20:00:27.377 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-10-05T20:00:27.542 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-10-05T20:00:27.565 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-10-05T20:00:27.589 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-10-05T20:00:27.657 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-10-05T20:00:28.328 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-10-05T20:00:28.404 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-10-05T20:00:28.508 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-10-05T20:00:28.637 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-10-05T20:00:28.764 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-10-05T20:00:28.852 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-10-05T20:00:28.933 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-10-05T20:00:29.112 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-10-05T20:00:29.167 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-10-05T20:00:29.281 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-10-05T20:00:29.422 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-10-05T20:00:29.651 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-10-05T20:00:29.839 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-10-05T20:00:29.871 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-10-05T20:00:29.904 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-10-05T20:00:30.476 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-10-05T20:00:30.621 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-10-05T20:00:31.655 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-10-05T20:00:31.692 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-10-05T20:00:31.754 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-10-05T20:00:32.036 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-10-05T20:00:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-10-05T20:00:32.392 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-10-05T20:00:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-10-05T20:00:34.125 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-10-05T20:00:35.683 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-10-05T20:00:39.954 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-10-05T20:00:40.781 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-10-05T20:00:42.209 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-10-05T20:00:42.347 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-10-05T20:00:42.689 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-10-05T20:00:43.039 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-10-05T20:00:43.226 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-10-05T20:00:43.303 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-10-05T20:00:43.640 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-10-05T20:00:43.792 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-10-05T20:00:43.880 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-10-05T20:00:43.969 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-10-05T20:00:43.998 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-10-05T20:00:44.123 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-10-05T20:00:47.245 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-10-05T20:00:50.294 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-10-05T20:00:53.339 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-10-05T20:00:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-10-05T20:00:53.583 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-10-05T20:00:53.761 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-10-05T20:00:54.042 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-10-05T20:00:54.183 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-10-05T20:00:54.428 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-10-05T20:00:54.847 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-10-05T20:00:54.967 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-10-05T20:00:55.235 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-10-05T20:00:55.701 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-10-05T20:00:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-10-05T20:00:56.008 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-10-05T20:00:56.286 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-10-05T20:00:56.370 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-10-05T20:01:02.133 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-10-05T20:01:02.215 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-10-05T20:01:02.292 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-10-05T20:01:02.369 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-10-05T20:01:02.454 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-10-05T20:01:02.696 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-10-05T20:01:08.812 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-10-05T20:01:09.163 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-10-05T20:01:09.388 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-10-05T20:01:09.699 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-10-05T20:01:10.049 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-10-05T20:01:12.423 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-10-05T20:01:15.527 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-10-05T20:01:28.003 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-10-05T20:01:29.509 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-10-05T20:01:30.233 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-10-05T20:01:31.045 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-10-05T20:01:31.333 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-10-05T20:01:31.634 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-10-05T20:01:31.776 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-10-05T20:01:32.060 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-10-05T20:01:32.267 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-10-05T20:01:32.735 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-10-05T20:01:32.773 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-10-05T20:01:32.852 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-10-05T20:01:32.930 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-10-05T20:01:32.961 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-10-05T20:01:32.994 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-10-05T20:01:33.116 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-10-05T20:01:33.158 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-10-05T20:01:33.194 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-10-05T20:01:33.245 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-10-05T20:01:33.286 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-10-05T20:01:33.320 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-10-05T20:01:33.362 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-10-05T20:01:33.409 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-10-05T20:01:33.447 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-10-05T20:01:33.448 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-10-05T20:01:33.529 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-10-05T20:01:33.611 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-10-05T20:01:33.669 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-10-05T20:01:33.746 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-10-05T20:01:33.792 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-10-05T20:01:33.869 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-10-05T20:01:33.954 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-10-05T20:01:34.027 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-10-05T20:01:34.103 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-10-05T20:01:34.171 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-10-05T20:01:34.242 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-10-05T20:01:35.165 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-10-05T20:01:35.765 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-10-05T20:01:35.806 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-10-05T20:01:35.860 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-10-05T20:01:35.944 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-10-05T20:01:36.013 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-10-05T20:01:36.042 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-10-05T20:01:36.963 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-10-05T20:01:37.487 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-10-05T20:01:37.998 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-10-05T20:01:38.036 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-10-05T20:01:38.115 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-10-05T20:01:38.192 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-10-05T20:01:38.246 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-10-05T20:01:38.331 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-10-05T20:01:38.419 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-10-05T20:01:38.467 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-10-05T20:01:38.513 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-10-05T20:01:38.576 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-10-05T20:01:38.639 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-10-05T20:01:38.787 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-10-05T20:01:38.947 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-10-05T20:01:39.124 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-10-05T20:01:39.258 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-10-05T20:01:39.434 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-10-05T20:01:39.601 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-10-05T20:01:39.787 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-10-05T20:01:39.961 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-10-05T20:01:40.131 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-10-05T20:01:40.356 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-10-05T20:01:40.576 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-10-05T20:01:40.690 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-10-05T20:01:40.727 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-10-05T20:01:40.854 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-10-05T20:01:40.992 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-10-05T20:01:41.177 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-10-05T20:01:41.362 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-10-05T20:01:41.629 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-10-05T20:01:43.386 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-10-05T20:01:43.559 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-10-05T20:01:43.777 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-10-05T20:01:43.950 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-10-05T20:01:44.227 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-10-05T20:01:44.399 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-10-05T20:01:44.494 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-10-05T20:01:44.691 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-10-05T20:01:44.723 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-10-05T20:01:44.760 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-10-05T20:01:44.809 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-10-05T20:01:44.855 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-10-05T20:01:44.955 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-10-05T20:01:45.043 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-10-05T20:01:45.300 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-10-05T20:01:45.389 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-10-05T20:01:45.465 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-10-05T20:01:45.575 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-10-05T20:01:45.640 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-10-05T20:01:45.707 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-10-05T20:01:45.825 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-10-05T20:01:45.892 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-10-05T20:01:45.960 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-10-05T20:01:46.033 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-10-05T20:01:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-10-05T20:01:46.170 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-10-05T20:01:46.242 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-10-05T20:01:46.365 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-10-05T20:01:46.489 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-10-05T20:01:46.569 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-10-05T20:01:46.643 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-10-05T20:01:46.762 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-10-05T20:01:46.887 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-10-05T20:01:47.001 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-10-05T20:01:47.098 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-10-05T20:01:47.220 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-10-05T20:01:47.326 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-10-05T20:01:47.498 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-10-05T20:01:47.654 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-10-05T20:01:47.705 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-10-05T20:01:47.781 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-10-05T20:01:47.861 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-10-05T20:01:47.999 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-10-05T20:01:48.079 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-10-05T20:01:48.188 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-10-05T20:01:48.266 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-10-05T20:01:48.354 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-10-05T20:01:48.439 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-10-05T20:01:48.501 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-10-05T20:01:48.581 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-10-05T20:01:48.706 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-10-05T20:01:48.837 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-10-05T20:01:48.969 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-10-05T20:01:49.000 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-10-05T20:01:49.084 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-10-05T20:01:49.160 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-10-05T20:01:49.242 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-10-05T20:01:49.321 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-10-05T20:01:49.399 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-10-05T20:01:49.454 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-10-05T20:01:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-10-05T20:01:49.735 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-10-05T20:01:49.903 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-10-05T20:01:49.982 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-10-05T20:01:50.172 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-10-05T20:01:51.017 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-10-05T20:01:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-10-05T20:01:51.136 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-10-05T20:01:51.221 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-10-05T20:01:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-10-05T20:01:52.275 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-10-05T20:01:53.217 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-10-05T20:01:53.273 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-10-05T20:01:53.697 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-10-05T20:01:54.260 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-10-05T20:01:54.534 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-10-05T20:01:54.745 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-10-05T20:01:54.795 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-10-05T20:01:55.339 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-10-05T20:01:55.703 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-10-05T20:01:56.220 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-10-05T20:01:56.541 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-10-05T20:01:56.816 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-10-05T20:01:57.006 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-10-05T20:01:57.168 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-10-05T20:01:57.371 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-10-05T20:01:57.642 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-10-05T20:01:57.731 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-10-05T20:01:57.953 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-10-05T20:01:58.167 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-10-05T20:01:58.400 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-10-05T20:01:58.661 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-10-05T20:01:58.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-10-05T20:01:58.876 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-10-05T20:01:59.240 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-10-05T20:01:59.620 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-10-05T20:01:59.877 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-10-05T20:02:00.074 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-10-05T20:02:00.257 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-10-05T20:02:13.324 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-10-05T20:02:13.324 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-10-05T20:03:13.159 INFO:teuthology.orchestra.run.smithi155.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-10-05T20:03:13.159 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T20:03:13.160 INFO:teuthology.orchestra.run.smithi155.stderr:---------------------------------------------------------------------- 2023-10-05T20:03:13.160 INFO:teuthology.orchestra.run.smithi155.stderr:Ran 618 tests in 223.616s 2023-10-05T20:03:13.160 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T20:03:13.160 INFO:teuthology.orchestra.run.smithi155.stderr:OK (SKIP=79) 2023-10-05T20:03:13.223 INFO:teuthology.orchestra.run.smithi155.stderr:Exception ignored in: .generate at 0x7f153700e620> 2023-10-05T20:03:13.223 INFO:teuthology.orchestra.run.smithi155.stderr:RuntimeError: generator ignored GeneratorExit 2023-10-05T20:03:13.252 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-10-05T20:03:13.252 DEBUG:teuthology.orchestra.run.smithi155:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-10-05T20:03:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:03:13.321 INFO:tasks.s3tests:Cleaning up boto... 2023-10-05T20:03:13.321 DEBUG:teuthology.orchestra.run.smithi155:> rm /home/ubuntu/cephtest/boto-client.0.cfg 2023-10-05T20:03:13.336 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-10-05T20:03:13.416 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:03:13.416 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:03:13.421 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.420+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.424 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.424 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 realm 2023-10-05T20:03:13.424 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.424 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.421+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.422+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.425 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.422+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.426 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.422+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:13.426 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.422+0000 7f6f2e03d680 20 searching for the correct realm 2023-10-05T20:03:13.431 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:13.431 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:13.431 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.432 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.430+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.431+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.431+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.431+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.433 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.431+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.434 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.431+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.441 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.437+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:13.442 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 zone default found 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 4 Realm: () 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T20:03:13.443 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 zonegroup default 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.438+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 started sync module instance, tier type = 2023-10-05T20:03:13.444 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.439+0000 7f6f2e03d680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T20:03:13.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=0 2023-10-05T20:03:13.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=1 2023-10-05T20:03:13.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=2 2023-10-05T20:03:13.472 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=3 2023-10-05T20:03:13.472 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=4 2023-10-05T20:03:13.472 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=5 2023-10-05T20:03:13.472 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.470+0000 7f6f2e03d680 20 add_watcher() i=6 2023-10-05T20:03:13.474 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.473+0000 7f6f2e03d680 20 add_watcher() i=7 2023-10-05T20:03:13.474 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.473+0000 7f6f2e03d680 2 all 8 watchers are set, enabling cache 2023-10-05T20:03:13.477 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.475+0000 7f6ef27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:13.477 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.475+0000 7f6f2e03d680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:03:13.477 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.475+0000 7f6f2e03d680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:03:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6e617fa700 20 reqs_thread_entry: start 2023-10-05T20:03:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6e5affd700 20 reqs_thread_entry: start 2023-10-05T20:03:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 20 init_complete bucket index max shards: 11 2023-10-05T20:03:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:03:13.540 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6e597fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:03:43 2023 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6e58ff9700 20 reqs_thread_entry: start 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 20 get_system_obj_state: rctx=0x7ffed416d858 obj=default.rgw.meta:users.uid:foo.client.0 state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 20 Read xattr: user.rgw.idtag 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.541 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.536+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 20 get_system_obj_state: rctx=0x7ffed416d510 obj=default.rgw.meta:root:.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.537+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 : miss 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 info.flags=0x16 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 adding default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 to cache LRU end 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.542 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 20 Read xattr: user.rgw.acl 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 20 Read xattr: user.rgw.idtag 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=364 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 info.flags=0x13 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 moving default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 to cache LRU end 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 2023-10-05T20:03:13.543 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.538+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T20:03:13.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 470F35 2023-10-05T20:03:13.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got DB8CE8 2023-10-05T20:03:13.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got FEC087 2023-10-05T20:03:13.545 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 4CCBC0 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got DE1AE8 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got F47C40 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 8A2DEA 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.539+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got A0E83A 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got C09C51 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 3E6BE8 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 11E507 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 36A3DB 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 521C92 2023-10-05T20:03:13.546 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 99DD45 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.540+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got E11836 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 3FB1EC 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got E2A2C5 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got F04888 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 695F2D 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 8A29C5 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.541+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got E3160D 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got A38938 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got CC1E4A 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got E0EF71 2023-10-05T20:03:13.547 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 122391 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got CD8E36 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 87F7F9 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got E8B039 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got EB3228 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.542+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:470F35 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.543+0000 7f6f2e03d680 10 manifest: total_size = 399218 2023-10-05T20:03:13.549 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.543+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.15548548503625984426 2023-10-05T20:03:13.549 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.543+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:470F35 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.549 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.543+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.549 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.543+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.0 2023-10-05T20:03:13.552 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-10-05T20:03:13.552 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.545+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:DB8CE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.554 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f2e03d680 10 manifest: total_size = 399324 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.7757624933131169677 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:DB8CE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.0 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.546+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=470F35 2023-10-05T20:03:13.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:FEC087 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 10 manifest: total_size = 399267 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.3762203275713059955 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:FEC087 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.548+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.0 2023-10-05T20:03:13.557 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.549+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=DB8CE8 2023-10-05T20:03:13.558 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.558 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.554+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:4CCBC0 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.555+0000 7f6f2e03d680 10 manifest: total_size = 398964 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.555+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.2622860042269384732 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.555+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:4CCBC0 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.555+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.555+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.1 2023-10-05T20:03:13.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.557+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=FEC087 2023-10-05T20:03:13.561 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:DE1AE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 10 manifest: total_size = 643224 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.16401537326447680825 2023-10-05T20:03:13.562 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:DE1AE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.563 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.563 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.560+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.1 2023-10-05T20:03:13.564 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.563+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=4CCBC0 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.566+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:F47C40 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.568 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.567+0000 7f6f2e03d680 10 manifest: total_size = 399118 2023-10-05T20:03:13.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.567+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.3281396878095897215 2023-10-05T20:03:13.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.567+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:F47C40 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.567+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.569 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.567+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.1 2023-10-05T20:03:13.570 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.569+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=DE1AE8 2023-10-05T20:03:13.574 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.574 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.574 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.572+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:8A2DEA state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.573+0000 7f6f2e03d680 10 manifest: total_size = 399221 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.573+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.14874638807280734837 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.573+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:8A2DEA state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.573+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.573+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.2 2023-10-05T20:03:13.575 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.574+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=F47C40 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.575+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:A0E83A state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.577 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f2e03d680 10 manifest: total_size = 399199 2023-10-05T20:03:13.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.8754992668157414264 2023-10-05T20:03:13.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:A0E83A state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.2 2023-10-05T20:03:13.578 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.576+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=8A2DEA 2023-10-05T20:03:13.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.580 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.578+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:C09C51 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.579+0000 7f6f2e03d680 10 manifest: total_size = 399103 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.579+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.3120968758694907771 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.579+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:C09C51 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.579+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.579+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.3 2023-10-05T20:03:13.581 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.580+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=A0E83A 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.584 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:3E6BE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.585 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 10 manifest: total_size = 180000 2023-10-05T20:03:13.585 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.12803001487736623074 2023-10-05T20:03:13.585 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:3E6BE8 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.585 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.585 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.582+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.4 2023-10-05T20:03:13.586 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.585+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=C09C51 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:11E507 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 10 manifest: total_size = 399247 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.17902167224944489 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:11E507 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.588 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.589 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.586+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.5 2023-10-05T20:03:13.589 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.587+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=3E6BE8 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:36A3DB state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 10 manifest: total_size = 117 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.12776213043094160360 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:36A3DB state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.592 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.590+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.5 2023-10-05T20:03:13.593 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.591+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=11E507 2023-10-05T20:03:13.594 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:521C92 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 10 manifest: total_size = 399185 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.7585051558344968420 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:521C92 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.595 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.593+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.5 2023-10-05T20:03:13.597 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.596+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=36A3DB 2023-10-05T20:03:13.599 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.597+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.599 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.597+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.599 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.597+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.599 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.597+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.599 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.597+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:99DD45 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 10 manifest: total_size = 39911527 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.12061535649121749807 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:99DD45 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.600 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.598+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.5 2023-10-05T20:03:13.601 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.600+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=521C92 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.609 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.610 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 RGWObjManifest::operator++(): result: ofs=39911527 stripe_ofs=39911527 part_ofs=0 rule->part_size=0 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - tag is: e34ad039-c923-4254-b20a-6f819b5e9a52.4205.12061535649121749807 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_1 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 333 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_2 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 565 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_3 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 797 2023-10-05T20:03:13.611 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_4 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1029 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_5 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1261 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_6 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1493 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_7 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1725 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_8 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1957 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - adding obj with name: e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518__shadow_.M08b9j0EeiTczyRmuWz352g5Hf5DYZx_9 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 2189 2023-10-05T20:03:13.612 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-10-05T20:03:13.613 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.606+0000 7f6f2e03d680 20 garbage collection: RGWGC::send_chain - on object name: gc.25tag is: e34ad039-c923-4254-b20a-6f819b5e9a52.4205.12061535649121749807 2023-10-05T20:03:13.613 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=99DD45 2023-10-05T20:03:13.613 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E11836 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.614 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 10 manifest: total_size = 5 2023-10-05T20:03:13.614 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.15581800754979844697 2023-10-05T20:03:13.614 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E11836 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.614 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.614 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.609+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.5 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:3FB1EC state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 10 manifest: total_size = 180000 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.7082229140702337816 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:3FB1EC state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.615 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.616 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.613+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.6 2023-10-05T20:03:13.617 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.615+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=E11836 2023-10-05T20:03:13.618 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E2A2C5 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 10 manifest: total_size = 399124 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.16941073343117020131 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E2A2C5 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.619 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.617+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.6 2023-10-05T20:03:13.620 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.619+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=3FB1EC 2023-10-05T20:03:13.624 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:F04888 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 10 manifest: total_size = 5 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.2073320315649228698 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:F04888 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.625 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.626 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.623+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.6 2023-10-05T20:03:13.628 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.627+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=E2A2C5 2023-10-05T20:03:13.632 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.632 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.632 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:695F2D state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 10 manifest: total_size = 399074 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.2725692135092388132 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:695F2D state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.633 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.631+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.7 2023-10-05T20:03:13.634 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.633+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=F04888 2023-10-05T20:03:13.638 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.638 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:8A29C5 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 10 manifest: total_size = 398909 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.4811352039885321973 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:8A29C5 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.639 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.637+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.7 2023-10-05T20:03:13.640 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.639+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=695F2D 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.644 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.642+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E3160D state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.643+0000 7f6f2e03d680 10 manifest: total_size = 399046 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.643+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.4429527363416218733 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.643+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E3160D state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.643+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.643+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.7 2023-10-05T20:03:13.645 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.644+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=8A29C5 2023-10-05T20:03:13.647 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.647 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.647 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:A38938 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 10 manifest: total_size = 399205 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.7082098669312315801 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:A38938 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.646+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.8 2023-10-05T20:03:13.648 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.647+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=E3160D 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.650 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.648+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:CC1E4A state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f2e03d680 10 manifest: total_size = 360236 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.14317007807910777984 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:CC1E4A state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.8 2023-10-05T20:03:13.651 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.649+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=A38938 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.653 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.651+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E0EF71 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f2e03d680 10 manifest: total_size = 643632 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.17579025682720118580 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E0EF71 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.8 2023-10-05T20:03:13.654 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.652+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=CC1E4A 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.656 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.654+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:122391 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.655+0000 7f6f2e03d680 10 manifest: total_size = 49186 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.655+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.3079680509105406448 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.655+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:122391 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.655+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.655+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.9 2023-10-05T20:03:13.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.656+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=E0EF71 2023-10-05T20:03:13.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.658 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:CD8E36 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 10 manifest: total_size = 575874 2023-10-05T20:03:13.659 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.1155381415372790777 2023-10-05T20:03:13.660 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:CD8E36 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.660 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.660 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.657+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.9 2023-10-05T20:03:13.660 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.658+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=122391 2023-10-05T20:03:13.662 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.662 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.662 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:87F7F9 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 10 manifest: total_size = 574264 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.15966064560316116137 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:87F7F9 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.661+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.10 2023-10-05T20:03:13.663 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.662+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=CD8E36 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E8B039 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.666 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 10 manifest: total_size = 590354 2023-10-05T20:03:13.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.14177362517137539389 2023-10-05T20:03:13.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:E8B039 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.664+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.10 2023-10-05T20:03:13.667 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.665+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=87F7F9 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:EB3228 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 10 manifest: total_size = 39 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.8992970028979122647 2023-10-05T20:03:13.671 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=test:EB3228 state=0x558c2ba56108 s->prefetch_data=0 2023-10-05T20:03:13.672 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2ba56108 2023-10-05T20:03:13.672 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.669+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518.10 2023-10-05T20:03:13.673 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.672+0000 7f6f127fc700 20 handle_completion(): completion ok for obj=E8B039 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T20:03:13.677 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T20:03:13.678 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.675+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=EB3228 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518])"; uploads.size()=0; is_truncated=0 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]) 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 lifecycle: get_system_obj_state: rctx=0x7ffed416cfe8 obj=default.rgw.meta:root:test state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-10-05T20:03:13.679 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.677+0000 7f6f2e03d680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T20:03:13.681 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.678+0000 7f6f2e03d680 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-10-05T20:03:13.681 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.678+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-10-05T20:03:13.681 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.678+0000 7f6f2e03d680 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-10-05T20:03:13.681 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.678+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 info.flags=0x17 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 to cache LRU end 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518, ofs0, ns] 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888961 cookie 94060514947808 notifier 4247 bl.length()=800 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f12ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 info.flags=0x17 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f12ffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 to cache LRU end 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f12ffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]): orig_sources= new_sources= 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]): potential sources added=[] removed=[] 2023-10-05T20:03:13.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]): orig_dests= new_dests= 2023-10-05T20:03:13.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.679+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:test[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518]): potential dests added=[] removed=[] 2023-10-05T20:03:13.686 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 20 get_system_obj_state: rctx=0x7ffed416ca48 obj=default.rgw.meta:root:test state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 moving default.rgw.meta+root+test to cache LRU end 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 removing default.rgw.meta+root+test from cache 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-10-05T20:03:13.687 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.685+0000 7f6f127fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888736 cookie 94060514952144 notifier 4247 bl.length()=148 2023-10-05T20:03:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.687+0000 7f6f2e03d680 10 removing default.rgw.meta+root+.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 from cache 2023-10-05T20:03:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.687+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518, ofs0, ns] 2023-10-05T20:03:13.688 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.687+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888962 cookie 94060514947808 notifier 4247 bl.length()=207 2023-10-05T20:03:13.694 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.693+0000 7f6f2e03d680 10 removing default.rgw.log++pubsub..bucket.test/e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518 from cache 2023-10-05T20:03:13.694 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.693+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/e34ad039-c923-4254-b20a-6f819b5e9a52.4207.518, ofs0, ns] 2023-10-05T20:03:13.695 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.693+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888737 cookie 94060514952144 notifier 4247 bl.length()=204 2023-10-05T20:03:13.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 20 get_system_obj_state: rctx=0x7ffed416d510 obj=default.rgw.meta:root:.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 : miss 2023-10-05T20:03:13.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 info.flags=0x16 2023-10-05T20:03:13.698 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 to cache LRU end 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 20 Read xattr: user.rgw.acl 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 20 Read xattr: user.rgw.idtag 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.696+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 info.flags=0x13 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 to cache LRU end 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.699 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.700 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 2023-10-05T20:03:13.700 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T20:03:13.700 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.697+0000 7f6f2e03d680 10 cls_bucket_list_unordered: got 61FB82 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=testbuck2:61FB82 state=0x558c2b9e1be8 s->prefetch_data=0 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 10 manifest: total_size = 399199 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 20 get_obj_state: setting s->obj_tag to e34ad039-c923-4254-b20a-6f819b5e9a52.4205.2929351225335299820 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 20 get_obj_state: rctx=0x7ffed416d520 obj=testbuck2:61FB82 state=0x558c2b9e1be8 s->prefetch_data=0 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 20 prepare_atomic_modification: state is not atomic. state=0x558c2b9e1be8 2023-10-05T20:03:13.702 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.700+0000 7f6f2e03d680 20 bucket index object: default.rgw.buckets.index:.dir.e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519.0 2023-10-05T20:03:13.704 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.702+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.702+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 chain_cache_entry: cache_locator= 2023-10-05T20:03:13.705 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 chain_cache_entry: couldn't find cache locator 2023-10-05T20:03:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T20:03:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T20:03:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T20:03:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.703+0000 7f6f2e03d680 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T20:03:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f12ffd700 20 handle_completion(): completion ok for obj=61FB82 2023-10-05T20:03:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519])"; uploads.size()=0; is_truncated=0 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]) 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 lifecycle: get_system_obj_state: rctx=0x7ffed416cfe8 obj=default.rgw.meta:root:testbuck2 state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.704+0000 7f6f2e03d680 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.705+0000 7f6f2e03d680 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.705+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.705+0000 7f6f2e03d680 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.705+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 info.flags=0x17 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 to cache LRU end 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519, ofs0, ns] 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888963 cookie 94060514947808 notifier 4247 bl.length()=810 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f12ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 info.flags=0x17 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f12ffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 to cache LRU end 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f12ffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]): orig_sources= new_sources= 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]): potential sources added=[] removed=[] 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]): orig_dests= new_dests= 2023-10-05T20:03:13.710 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.706+0000 7f6f2e03d680 20 lifecycle: handle_bi_update(): bucket=:testbuck2[e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519]): potential dests added=[] removed=[] 2023-10-05T20:03:13.716 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.714+0000 7f6f2e03d680 20 get_system_obj_state: rctx=0x7ffed416ca48 obj=default.rgw.meta:root:testbuck2 state=0x558c2b7bb240 s->prefetch_data=0 2023-10-05T20:03:13.716 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.714+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-10-05T20:03:13.716 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-10-05T20:03:13.716 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 removing default.rgw.meta+root+testbuck2 from cache 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-10-05T20:03:13.717 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.715+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888738 cookie 94060514944624 notifier 4247 bl.length()=153 2023-10-05T20:03:13.718 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.716+0000 7f6f2e03d680 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 from cache 2023-10-05T20:03:13.718 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.716+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519, ofs0, ns] 2023-10-05T20:03:13.718 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.716+0000 7f6f127fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888964 cookie 94060514947808 notifier 4247 bl.length()=212 2023-10-05T20:03:13.727 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.726+0000 7f6f2e03d680 10 removing default.rgw.log++pubsub..bucket.testbuck2/e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519 from cache 2023-10-05T20:03:13.727 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.726+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/e34ad039-c923-4254-b20a-6f819b5e9a52.4207.519, ofs0, ns] 2023-10-05T20:03:13.728 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.726+0000 7f6f127fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888739 cookie 94060514952144 notifier 4247 bl.length()=209 2023-10-05T20:03:13.730 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.729+0000 7f6f2e03d680 10 removing key index: KGAWZTWMIAQUYPOGNDEV 2023-10-05T20:03:13.731 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.729+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:KGAWZTWMIAQUYPOGNDEV, ofs0, ns] 2023-10-05T20:03:13.731 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.729+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888965 cookie 94060514931728 notifier 4247 bl.length()=170 2023-10-05T20:03:13.732 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.730+0000 7f6f2e03d680 10 removing email index: foo.client.0+test@test.test 2023-10-05T20:03:13.732 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.730+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-10-05T20:03:13.732 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.731+0000 7f6f127fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888740 cookie 94060514952144 notifier 4247 bl.length()=178 2023-10-05T20:03:13.733 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.732+0000 7f6f2e03d680 10 removing user buckets index 2023-10-05T20:03:13.733 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.732+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-10-05T20:03:13.734 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.732+0000 7f6f12ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888741 cookie 94060514935040 notifier 4247 bl.length()=169 2023-10-05T20:03:13.735 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.734+0000 7f6f2e03d680 10 removing user index: foo.client.0 2023-10-05T20:03:13.736 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.734+0000 7f6f2e03d680 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-10-05T20:03:13.736 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.734+0000 7f6f2e03d680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-10-05T20:03:13.736 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.734+0000 7f6f127fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888966 cookie 94060514931728 notifier 4247 bl.length()=161 2023-10-05T20:03:13.739 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.738+0000 7f6f2e03d680 20 remove_watcher() i=0 2023-10-05T20:03:13.739 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.738+0000 7f6f2e03d680 2 removed watcher, disabling cache 2023-10-05T20:03:13.740 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.739+0000 7f6f2e03d680 20 remove_watcher() i=1 2023-10-05T20:03:13.742 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.740+0000 7f6f2e03d680 20 remove_watcher() i=2 2023-10-05T20:03:13.744 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.743+0000 7f6f2e03d680 20 remove_watcher() i=3 2023-10-05T20:03:13.746 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.745+0000 7f6f2e03d680 20 remove_watcher() i=4 2023-10-05T20:03:13.747 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.746+0000 7f6f2e03d680 20 remove_watcher() i=5 2023-10-05T20:03:13.749 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.748+0000 7f6f2e03d680 20 remove_watcher() i=6 2023-10-05T20:03:13.750 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.748+0000 7f6f2e03d680 20 remove_watcher() i=7 2023-10-05T20:03:13.750 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.749+0000 7f6ef27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:13.756 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-10-05T20:03:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:03:13.835 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:03:13.840 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.839+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.842 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.842 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 realm 2023-10-05T20:03:13.842 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.842 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.840+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.841+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.843 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.841+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.844 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.841+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:13.844 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.841+0000 7f118820e680 20 searching for the correct realm 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.851 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.848+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.852 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.853 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.849+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.858 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:13.858 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:13.858 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.855+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.859 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 zone default found 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 4 Realm: () 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 zonegroup default 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.856+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.860 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:13.861 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 started sync module instance, tier type = 2023-10-05T20:03:13.861 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.857+0000 7f118820e680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=0 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=1 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=2 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=3 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=4 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=5 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.866+0000 7f118820e680 20 add_watcher() i=6 2023-10-05T20:03:13.868 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.867+0000 7f118820e680 20 add_watcher() i=7 2023-10-05T20:03:13.869 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.867+0000 7f118820e680 2 all 8 watchers are set, enabling cache 2023-10-05T20:03:13.870 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.869+0000 7f114cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:13.870 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.869+0000 7f118820e680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:03:13.870 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.869+0000 7f118820e680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:03:13.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f10bb7fe700 20 reqs_thread_entry: start 2023-10-05T20:03:13.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f10b97fa700 20 reqs_thread_entry: start 2023-10-05T20:03:13.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f118820e680 20 init_complete bucket index max shards: 11 2023-10-05T20:03:13.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f118820e680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:03:13.918 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f10a77fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:03:44 2023 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f10a6ffd700 20 reqs_thread_entry: start 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f118820e680 20 get_system_obj_state: rctx=0x7ffeb8187ad8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55c780fdf240 s->prefetch_data=0 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.915+0000 7f118820e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 20 Read xattr: user.rgw.idtag 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:13.919 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 20 rados->read ofs=0 len=0 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 20 rados_obj.operate() r=0 bl.length=469 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 removing key index: LFWOHPXQWFGHREZAHFEI 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.916+0000 7f118820e680 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.keys:LFWOHPXQWFGHREZAHFEI, ofs0, ns] 2023-10-05T20:03:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.917+0000 7f1179b05700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888967 cookie 94315352600032 notifier 4253 bl.length()=170 2023-10-05T20:03:13.921 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.918+0000 7f118820e680 10 removing email index: bar.client.0+test@test.test 2023-10-05T20:03:13.921 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.918+0000 7f118820e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-10-05T20:03:13.921 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.918+0000 7f1179304700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888742 cookie 94315352601456 notifier 4253 bl.length()=178 2023-10-05T20:03:13.921 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.919+0000 7f118820e680 10 removing user buckets index 2023-10-05T20:03:13.921 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.919+0000 7f118820e680 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-10-05T20:03:13.922 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.920+0000 7f1179b05700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028889781 cookie 94315352604064 notifier 4253 bl.length()=169 2023-10-05T20:03:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.922+0000 7f118820e680 10 removing user index: bar.client.0 2023-10-05T20:03:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.922+0000 7f118820e680 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-10-05T20:03:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.922+0000 7f118820e680 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-10-05T20:03:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.922+0000 7f1179304700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888743 cookie 94315352601456 notifier 4253 bl.length()=161 2023-10-05T20:03:13.929 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.928+0000 7f118820e680 20 remove_watcher() i=0 2023-10-05T20:03:13.929 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.928+0000 7f118820e680 2 removed watcher, disabling cache 2023-10-05T20:03:13.930 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.928+0000 7f118820e680 20 remove_watcher() i=1 2023-10-05T20:03:13.933 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.931+0000 7f118820e680 20 remove_watcher() i=2 2023-10-05T20:03:13.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.934+0000 7f118820e680 20 remove_watcher() i=3 2023-10-05T20:03:13.941 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.939+0000 7f118820e680 20 remove_watcher() i=4 2023-10-05T20:03:13.941 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.940+0000 7f118820e680 20 remove_watcher() i=5 2023-10-05T20:03:13.942 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.941+0000 7f118820e680 20 remove_watcher() i=6 2023-10-05T20:03:13.943 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.942+0000 7f118820e680 20 remove_watcher() i=7 2023-10-05T20:03:13.944 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:13.943+0000 7f114cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:13.949 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-10-05T20:03:14.030 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setuser ceph since I am not root 2023-10-05T20:03:14.030 INFO:teuthology.orchestra.run.smithi155.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T20:03:14.035 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.034+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.037 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.037 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 realm 2023-10-05T20:03:14.037 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.037 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.037 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.035+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.036+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.036+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.038 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.036+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:14.039 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.036+0000 7ff6a01bc680 20 searching for the correct realm 2023-10-05T20:03:14.045 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:14.045 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.043+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.046 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.047 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.047 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.044+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:14.052 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zonegroup_info.0d58a1a5-7f52-4137-974a-dc60a3733dd8 2023-10-05T20:03:14.052 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zone_info.e34ad039-c923-4254-b20a-6f819b5e9a52 2023-10-05T20:03:14.052 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.050+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T20:03:14.053 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.054 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 zone default found 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 4 Realm: () 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 4 ZoneGroup: default (0d58a1a5-7f52-4137-974a-dc60a3733dd8) 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 4 Zone: default (e34ad039-c923-4254-b20a-6f819b5e9a52) 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 zonegroup default 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.051+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.052+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.055 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.052+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.056 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.052+0000 7ff6a01bc680 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T20:03:14.056 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.052+0000 7ff6a01bc680 20 started sync module instance, tier type = 2023-10-05T20:03:14.056 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.052+0000 7ff6a01bc680 20 started zone id=e34ad039-c923-4254-b20a-6f819b5e9a52 (name=default) with tier type = 2023-10-05T20:03:14.079 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=0 2023-10-05T20:03:14.079 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=1 2023-10-05T20:03:14.079 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=2 2023-10-05T20:03:14.079 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=3 2023-10-05T20:03:14.080 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=4 2023-10-05T20:03:14.080 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=5 2023-10-05T20:03:14.080 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.078+0000 7ff6a01bc680 20 add_watcher() i=6 2023-10-05T20:03:14.080 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.079+0000 7ff6a01bc680 20 add_watcher() i=7 2023-10-05T20:03:14.080 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.079+0000 7ff6a01bc680 2 all 8 watchers are set, enabling cache 2023-10-05T20:03:14.082 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.080+0000 7ff5dffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:14.082 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.080+0000 7ff6a01bc680 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T20:03:14.082 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.080+0000 7ff6a01bc680 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.129+0000 7ff5ceffd700 20 reqs_thread_entry: start 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.129+0000 7ff5ccff9700 20 reqs_thread_entry: start 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.129+0000 7ff6a01bc680 20 init_complete bucket index max shards: 11 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff5c6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 20:03:44 2023 2023-10-05T20:03:14.132 INFO:teuthology.orchestra.run.smithi155.stderr: 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff5c67fc700 20 reqs_thread_entry: start 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 20 get_system_obj_state: rctx=0x7fff27b3a368 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55d53a71a240 s->prefetch_data=0 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 20 Read xattr: user.rgw.idtag 2023-10-05T20:03:14.133 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 20 rados->read ofs=0 len=0 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 20 rados_obj.operate() r=0 bl.length=528 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.130+0000 7ff6a01bc680 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.131+0000 7ff6a01bc680 10 removing key index: XCWEZXIPPXCDUXORUFKW 2023-10-05T20:03:14.134 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.131+0000 7ff6a01bc680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:XCWEZXIPPXCDUXORUFKW, ofs0, ns] 2023-10-05T20:03:14.135 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.131+0000 7ff6912b2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888968 cookie 94374298557984 notifier 4259 bl.length()=170 2023-10-05T20:03:14.135 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.132+0000 7ff6a01bc680 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-10-05T20:03:14.135 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.132+0000 7ff6a01bc680 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-10-05T20:03:14.136 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.132+0000 7ff6912b2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888969 cookie 94374298554784 notifier 4259 bl.length()=193 2023-10-05T20:03:14.136 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.134+0000 7ff6a01bc680 10 removing user buckets index 2023-10-05T20:03:14.136 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.134+0000 7ff6a01bc680 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-10-05T20:03:14.136 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.134+0000 7ff690ab1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888744 cookie 94374298547824 notifier 4259 bl.length()=184 2023-10-05T20:03:14.137 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.135+0000 7ff6a01bc680 10 removing user index: testx$tenanteduser.client.0 2023-10-05T20:03:14.137 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.135+0000 7ff6a01bc680 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-10-05T20:03:14.137 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.135+0000 7ff6a01bc680 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-10-05T20:03:14.137 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.135+0000 7ff6912b2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888970 cookie 94374298557984 notifier 4259 bl.length()=176 2023-10-05T20:03:14.139 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.138+0000 7ff6a01bc680 20 remove_watcher() i=0 2023-10-05T20:03:14.139 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.138+0000 7ff6a01bc680 2 removed watcher, disabling cache 2023-10-05T20:03:14.140 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.139+0000 7ff6a01bc680 20 remove_watcher() i=1 2023-10-05T20:03:14.144 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.142+0000 7ff6a01bc680 20 remove_watcher() i=2 2023-10-05T20:03:14.144 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.143+0000 7ff6a01bc680 20 remove_watcher() i=3 2023-10-05T20:03:14.147 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.146+0000 7ff6a01bc680 20 remove_watcher() i=4 2023-10-05T20:03:14.150 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.149+0000 7ff6a01bc680 20 remove_watcher() i=5 2023-10-05T20:03:14.151 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.150+0000 7ff6a01bc680 20 remove_watcher() i=6 2023-10-05T20:03:14.153 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.152+0000 7ff6a01bc680 20 remove_watcher() i=7 2023-10-05T20:03:14.154 INFO:teuthology.orchestra.run.smithi155.stderr:2023-10-05T20:03:14.152+0000 7ff5dffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T20:03:14.160 INFO:tasks.s3tests:Removing s3-tests... 2023-10-05T20:03:14.160 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/s3-tests-client.0 2023-10-05T20:03:14.320 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-10-05T20:03:14.330 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T20:03:14.355 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T20:03:14.356 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:14.356 INFO:tasks.rgw.client.0.smithi155.stdout:2023-10-05T20:03:14.355+0000 7f2bf09b3700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi155.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --rgw_crypt_sse_s3_vault_addr smithi155.front.sepia.ceph.com:8200 --rgw_crypt_sse_s3_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 90681) UID: 0 2023-10-05T20:03:14.357 INFO:tasks.rgw.client.0.smithi155.stdout:2023-10-05T20:03:14.355+0000 7f2c01c8e740 -1 shutting down 2023-10-05T20:03:17.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T20:03:20.458 INFO:tasks.rgw.client.0:Stopped 2023-10-05T20:03:20.458 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T20:03:20.476 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T20:03:20.557 DEBUG:teuthology.run_tasks:Unwinding manager vault 2023-10-05T20:03:20.568 INFO:tasks.vault:Stopping Vault instance 2023-10-05T20:03:20.568 DEBUG:tasks.vault.client.0:waiting for process to exit 2023-10-05T20:03:20.568 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:20.599 INFO:tasks.vault.client.0.smithi155.stdout:==> Vault shutdown triggered 2023-10-05T20:03:20.599 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.597Z [INFO] core: marked as sealed 2023-10-05T20:03:20.600 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.598Z [INFO] core: pre-seal teardown starting 2023-10-05T20:03:20.600 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.598Z [INFO] rollback: stopping rollback manager 2023-10-05T20:03:20.600 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.598Z [INFO] core: pre-seal teardown complete 2023-10-05T20:03:20.600 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.598Z [INFO] core: stopping cluster listeners 2023-10-05T20:03:20.600 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.598Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2023-10-05T20:03:20.629 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.628Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2023-10-05T20:03:20.629 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.628Z [INFO] core: cluster listeners successfully shut down 2023-10-05T20:03:20.629 INFO:tasks.vault.client.0.smithi155.stderr:2023-10-05T20:03:20.628Z [INFO] core: vault is sealed 2023-10-05T20:03:20.630 INFO:tasks.vault.client.0:Stopped 2023-10-05T20:03:20.630 INFO:tasks.vault:Removing Vault... 2023-10-05T20:03:20.631 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2023-10-05T20:03:20.703 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T20:03:20.712 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T20:03:20.712 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T20:03:21.053 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-10-05T20:03:21.054 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":34,"fsid":"37cff2ea-3635-4882-b7b9-02c09cc5110d","created":"2023-10-05T19:58:05.352962+0000","modified":"2023-10-05T20:00:12.516736+0000","last_up_change":"2023-10-05T19:58:12.919526+0000","last_in_change":"2023-10-05T19:58:06.300900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:58:13.162201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:58:13.977145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T19:58:31.383101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T19:58:33.694510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T19:58:35.813426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T19:58:37.418984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T19:58:39.410377+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T19:58:41.478168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-10-05T19:58:51.938651+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-10-05T19:59:30.854118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"07531411-8f81-469c-81ef-be6684f9f9d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":85992},{"type":"v1","addr":"172.21.15.155:6817","nonce":85992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":85992},{"type":"v1","addr":"172.21.15.155:6819","nonce":85992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":85992},{"type":"v1","addr":"172.21.15.155:6823","nonce":85992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":85992},{"type":"v1","addr":"172.21.15.155:6821","nonce":85992}]},"public_addr":"172.21.15.155:6817/85992","cluster_addr":"172.21.15.155:6819/85992","heartbeat_back_addr":"172.21.15.155:6823/85992","heartbeat_front_addr":"172.21.15.155:6821/85992","state":["exists","up"]},{"osd":1,"uuid":"e8640ac5-0586-4a42-9b90-0467c565203f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":86001},{"type":"v1","addr":"172.21.15.155:6809","nonce":86001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":86001},{"type":"v1","addr":"172.21.15.155:6811","nonce":86001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":86001},{"type":"v1","addr":"172.21.15.155:6815","nonce":86001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":86001},{"type":"v1","addr":"172.21.15.155:6813","nonce":86001}]},"public_addr":"172.21.15.155:6809/86001","cluster_addr":"172.21.15.155:6811/86001","heartbeat_back_addr":"172.21.15.155:6815/86001","heartbeat_front_addr":"172.21.15.155:6813/86001","state":["exists","up"]},{"osd":2,"uuid":"1a698c1f-172f-4092-a286-167bef22592a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":85985},{"type":"v1","addr":"172.21.15.155:6801","nonce":85985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":85985},{"type":"v1","addr":"172.21.15.155:6803","nonce":85985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":85985},{"type":"v1","addr":"172.21.15.155:6807","nonce":85985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":85985},{"type":"v1","addr":"172.21.15.155:6805","nonce":85985}]},"public_addr":"172.21.15.155:6801/85985","cluster_addr":"172.21.15.155:6803/85985","heartbeat_back_addr":"172.21.15.155:6807/85985","heartbeat_front_addr":"172.21.15.155:6805/85985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.670100+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.645307+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:58:10.703975+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":0,"to":1}]},{"pgid":"3.1c","mappings":[{"from":0,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"4.28","mappings":[{"from":1,"to":0}]},{"pgid":"4.39","mappings":[{"from":1,"to":2}]},{"pgid":"4.3a","mappings":[{"from":1,"to":0}]},{"pgid":"4.3d","mappings":[{"from":1,"to":2}]},{"pgid":"10.2","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T20:03:21.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T20:03:21.392 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T20:03:21.393 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T20:03:21.393 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T20:03:21.393 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:21.441 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T20:03:21.442 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T20:03:21.442 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:21.489 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T20:03:21.489 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T20:03:21.489 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:21.533 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T20:03:21.533 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T20:03:21.533 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T20:03:21.533 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:21.557 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T20:03:21.557 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T20:03:21.557 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T20:03:21.557 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T20:03:21.608 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T20:03:21.608 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T20:03:21.608 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:03:21.674 INFO:teuthology.orchestra.run.smithi155.stdout:1696535895.9281237 mon.a (mon.0) 83 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T20:03:21.675 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T20:03:21.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:03:21.741 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:03:21.807 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T20:03:21.873 INFO:teuthology.orchestra.run.smithi155.stdout:1696535895.9281237 mon.a (mon.0) 83 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T20:03:21.873 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T20:03:21.873 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T20:03:22.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T20:03:24.672 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T20:03:24.673 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T20:03:24.772 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T20:03:24.773 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T20:03:24.849 INFO:tasks.ceph:Archiving mon data... 2023-10-05T20:03:24.850 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769/data/mon.a.tgz 2023-10-05T20:03:24.851 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2023-10-05T20:03:24.865 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.ZqeJnqi6Yb 2023-10-05T20:03:24.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ZqeJnqi6Yb 2023-10-05T20:03:25.000 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.ZqeJnqi6Yb 2023-10-05T20:03:25.080 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.ZqeJnqi6Yb is 279KB 2023-10-05T20:03:25.175 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.ZqeJnqi6Yb 2023-10-05T20:03:25.189 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T20:03:25.189 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T20:03:25.251 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T20:03:25.252 DEBUG:teuthology.orchestra.run.smithi155:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T20:03:25.317 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T20:03:25.322 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T20:03:25.324 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769/remote/smithi155/crash 2023-10-05T20:03:25.325 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T20:03:25.387 INFO:tasks.ceph:Compressing logs... 2023-10-05T20:03:25.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T20:03:53.374 INFO:tasks.ceph:Archiving logs... 2023-10-05T20:03:53.374 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769/remote/smithi155/log 2023-10-05T20:03:53.376 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T20:03:56.621 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T20:03:57.072 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T20:03:57.072 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T20:03:57.120 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T20:03:57.129 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T20:03:57.130 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T20:03:57.153 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:210 Number of sources = 4 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -947us[ -947us] +/- 75ms 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -527us[ -527us] +/- 78ms 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv03.front.sepia.ceph.com 3 6 377 25 +1216us[+1216us] +/- 68ms 2023-10-05T20:03:57.157 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-05T20:03:57.158 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T20:03:57.168 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T20:03:57.168 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T20:03:57.182 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-05T20:03:57.288 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-05T20:03:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:03:57.416 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T20:03:57.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T20:03:57.447 INFO:teuthology.task.internal:Duration was 1092.963939 seconds 2023-10-05T20:03:57.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T20:03:57.466 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T20:03:57.466 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T20:03:57.510 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-05T20:03:57.723 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T20:03:57.723 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2023-10-05T20:03:57.723 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T20:03:57.742 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T20:03:57.742 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T20:03:57.820 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T20:03:57.820 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T20:03:57.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T20:03:57.944 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T20:03:57.944 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T20:03:57.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T20:03:57.979 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T20:03:58.032 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2023-10-05T20:03:58.152 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T20:03:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T20:03:58.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T20:03:58.217 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T20:03:58.218 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769/remote/smithi155 2023-10-05T20:03:58.218 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T20:03:58.282 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T20:03:58.282 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T20:03:58.329 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T20:03:58.343 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T20:03:58.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T20:03:58.357 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T20:03:58.357 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T20:03:58.383 INFO:teuthology.orchestra.run.smithi155.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 20:03 /home/ubuntu/cephtest 2023-10-05T20:03:58.384 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T20:03:58.395 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T20:03:58.430 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T20:03:58.456 DEBUG:teuthology.nuke:shortname: smithi155 2023-10-05T20:03:58.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T20:03:58.478 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411769', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:37:44.724970', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYiwqWwZ6dyEyEwETysEVt7X3DeM0X0sKB7EP5gkcGqxrvp1es/k5VfSSPvll5vzEXXKIz+V56EoZwhgmO4OoI='} 2023-10-05T20:03:58.528 INFO:teuthology.orchestra.console.smithi155:Power off 2023-10-05T20:03:58.528 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T20:03:58.553 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2023-10-05T20:03:58.554 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:03:58.575 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2023-10-05T20:04:02.576 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:04:02.686 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2023-10-05T20:04:06.690 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:04:06.839 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2023-10-05T20:04:10.882 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T20:04:10.992 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2023-10-05T20:04:11.092 INFO:teuthology.orchestra.console.smithi155:Power off completed 2023-10-05T20:04:11.196 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} duration: 1092.9639387130737 failure_reason: '"1696535895.9281237 mon.a (mon.0) 83 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T20:04:11.197 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T20:04:11.269 INFO:teuthology.run:FAIL